[Twisted-Python] Large Transfers
glyph at twistedmatrix.com
Sat May 10 11:46:33 EDT 2003
-----BEGIN PGP SIGNED MESSAGE-----
On Saturday, May 10, 2003, at 10:33 AM, Uwe C. Schroeder wrote:
>> That's *way* to DWIMy, IMHO.
> but it would be more convenient and transparent for the programmer not
> to have take care of paging :-)
And surprising in corner cases, and very difficult to debug. This is
basically the same case for using large numbers of threads rather than
an event-driven API. This is not how Twisted works.
In fact, we have taken great care to describe PB as a "translucent"
remote access API, never "transparent". Transparency is explicitly not
>> Use StringPager. It's in memory. *Always* use StringPager, even if
>> you're below the security limit.
> The busines logic behind it simply gets way to complicated if I have
> to separate calls into "small" and "large" ones.
Your response implies you have inverted Moshe's meaning. I think what
Moshe was saying, in your terminology, is: 'Do not separate calls into
"small" and "large" ones. Treat all calls as "large". In the lowere
levels of Twisted, this will do a reasonably efficient thing even if
the data is actually small, and it will not complicate your code.'
> if I have to split the stuff and put some logic in there that catches
> the large calls and pages them it will add another 5k lines of code.
Given the expansion that we have repeatedly given using
twisted.spread.util (this may turn 1 line into 3 lines, although in
most cases it just makes a slightly longer line) this implies that you
have at least 1600 separate lines where you are doing a callRemote that
is directly retrieving a potentially large XML payload. It seems like
this is something which you could get quite a bit of benefit out of
abstracting into a thinner interface.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (Darwin)
-----END PGP SIGNATURE-----
More information about the Twisted-Python