[Twisted-Python] Large Transfers

Uwe C. Schroeder uwe at oss4u.com
Sat May 10 09:59:16 EDT 2003


Hi again,

in  another thread we had the "pb size limitation" issue. However it still 
doesn't make too much sense to me, so probably someone of the core developers 
can enlighten me.
Assume I have a remotecall using pb. According to the size limitation the 
parameters given to a specific function can not exceed 640kb. What if I hand 
over an object that exceeds the 640k ? Ok, the obvious happens and cBanana 
throws an error "security precaution ....."
Is this really a good thing to do ? Shouldn't pb see that the arguments are 
larger than 640k and start paging ?
What I'm doing is to hand down XML data which is database-generated on the 
server side. Whenever a user requests a too large resultset the network layer 
fails. On the other hand the resultset already is in memory, so why not jut 
transfer it ?
I realize that his is probably bad design, but it's the easiest way to 
transfer this information. Sure I can write the stuff to a temporary file and 
page it over, however this defies the purpose, since then the original call 
results in a message to go get the file. This means I need at least 4 
callbacks for any given call (the original ok callback, then another one for 
the possible paging as well as 2 error callbacks, one for each call)
I can extend the problem by compressing the parameters with zlib ( which I'm 
doing anyways), but at some point I will hit the limit.
The other problem this creates is a timing issue. Since I have to make several 
calls in order to transfer the resultset, I have to delay database calls 
until the whole resultset is valid.
To put it more technically:

self.perspective.transfer_large_result(small_int,label,large_result_array)

will fail if large_result_array exceeds 640k. However small_int and label can 
be transferred. The only way to do this is

if numberofbytes(large_result_set) >= 640k:
	
self.perspective.transfer_first_part(small_int,label).callback(self.smallpart_ok)

and in smallpart_ok

self.page_the_rest(large_result_array).callback(self.whole_stuff_transfered)

I can continue the flow of the application in whole_stuff_transfered, however 
I have to temporarily store the values of small_int and label, since I can't 
create a database record without the rest.

This is an enormous overhead.

What even strikes me more is that this size limit not even prevents large 
memory consumption - since the object is already there and in cBanana the 
object is already stored in the buffer.

So what makes is "more secure" to limit the size ??

I think I'll just remove all size limits and go thru the (unwanted) way to 
create a own package.


	UC

--
Open Source Solutions 4U, LLC	2570 Fleetwood Drive
Phone:  +1 650 872 2425		San Bruno, CA 94066
Cell:   +1 650 302 2405		United States
Fax:    +1 650 872 2417





More information about the Twisted-Python mailing list