[Twisted-Python] Large Transfers
Uwe C. Schroeder
uwe at oss4u.com
Sat May 10 09:59:16 EDT 2003
in another thread we had the "pb size limitation" issue. However it still
doesn't make too much sense to me, so probably someone of the core developers
can enlighten me.
Assume I have a remotecall using pb. According to the size limitation the
parameters given to a specific function can not exceed 640kb. What if I hand
over an object that exceeds the 640k ? Ok, the obvious happens and cBanana
throws an error "security precaution ....."
Is this really a good thing to do ? Shouldn't pb see that the arguments are
larger than 640k and start paging ?
What I'm doing is to hand down XML data which is database-generated on the
server side. Whenever a user requests a too large resultset the network layer
fails. On the other hand the resultset already is in memory, so why not jut
transfer it ?
I realize that his is probably bad design, but it's the easiest way to
transfer this information. Sure I can write the stuff to a temporary file and
page it over, however this defies the purpose, since then the original call
results in a message to go get the file. This means I need at least 4
callbacks for any given call (the original ok callback, then another one for
the possible paging as well as 2 error callbacks, one for each call)
I can extend the problem by compressing the parameters with zlib ( which I'm
doing anyways), but at some point I will hit the limit.
The other problem this creates is a timing issue. Since I have to make several
calls in order to transfer the resultset, I have to delay database calls
until the whole resultset is valid.
To put it more technically:
will fail if large_result_array exceeds 640k. However small_int and label can
be transferred. The only way to do this is
if numberofbytes(large_result_set) >= 640k:
and in smallpart_ok
I can continue the flow of the application in whole_stuff_transfered, however
I have to temporarily store the values of small_int and label, since I can't
create a database record without the rest.
This is an enormous overhead.
What even strikes me more is that this size limit not even prevents large
memory consumption - since the object is already there and in cBanana the
object is already stored in the buffer.
So what makes is "more secure" to limit the size ??
I think I'll just remove all size limits and go thru the (unwanted) way to
create a own package.
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: +1 650 872 2425 San Bruno, CA 94066
Cell: +1 650 302 2405 United States
Fax: +1 650 872 2417
More information about the Twisted-Python