[Twisted-Python] Large Transfers

[Uwe C. Schroeder]

On Saturday 10 May 2003 08:15 am, Moshe Zadka wrote:
> On Sat, 10 May 2003, "Uwe C. Schroeder" <uwe at oss4u.com> wrote:
> > in  another thread we had the "pb size limitation" issue. However it
> > still doesn't make too much sense to me, so probably someone of the core
> > developers can enlighten me.
>
> The explanation is that the *other side* doesn't trust you -- maybe you're
> trying to DoS it? This is a good design decision: you want multiple
> processes to not trust each other. As an aside, whenever you hear the word
> "trust" you should think "lack of security" -- by extension, "distrust" is
> "more secure".

agreed

> > Assume I have a remotecall using pb. According to the size limitation the
> > parameters given to a specific function can not exceed 640kb. What if I
> > hand over an object that exceeds the 640k ? Ok, the obvious happens and
> > cBanana throws an error "security precaution ....."
>
> Yep. the cBanana on the other side decided that you looked like an evil
> DoSer, and terminated you.
>
> > Is this really a good thing to do ? Shouldn't pb see that the arguments
> > are larger than 640k and start paging ?
>
> That's *way* to DWIMy, IMHO.

but it would be more convenient and transparent for the programmer not to have 
take care of paging :-)

> > What I'm doing is to hand down XML data which is database-generated on
> > the server side. Whenever a user requests a too large resultset the
> > network layer fails. On the other hand the resultset already is in
> > memory, so why not jut transfer it ?
>
> Use StringPager. It's in memory. *Always* use StringPager, even if you're
> below the security limit. 8k (by default) things will get transferred
> in one go, larger strings will be sent in 8k chunks, allowing the other
> side to digest them slowly without building big *intermediary* structures
> in memory.

The busines logic behind it simply gets way to complicated if I have to 
separate calls into "small" and "large" ones. It's complicated already ( 
approaching 100k lines of code) and if I have to split the stuff and put some 
logic in there that catches the large calls and pages them it will add 
another 5k lines of code.

>
> > So what makes is "more secure" to limit the size ??
>
> That if your db-using process gets 0wned (if you pardon my trust),
> it won't be able to drag down the other process.

Well, unlikely but not a bad asumption. Better have it safe.
Since my application will run inside a trusted environment only (behind 
firewalls in an intranet) I think I'll just remove the size limit. Any 
connection from the outside will have to use VPN anyways, since I'm a 
distrusting person and won't let anyone into "my" network.

	UC

--
Open Source Solutions 4U, LLC	2570 Fleetwood Drive
Phone:  +1 650 872 2425		San Bruno, CA 94066
Cell:   +1 650 302 2405		United States
Fax:    +1 650 872 2417