[Twisted-web] t.w.client.downloadPage with large amount of postData

[exarkun at twistedmatrix.com]

On 5 Oct, 06:45 pm, seadog at sealabs.net wrote:
>Hello everyone,
>I'm writing a client that is using t.w.client.downloadPage function to
>download a large chunk of data into a file from a webserver. This is
>working fine.
>
>Now I want use the same function (or at least take advantage of the
>"save to file" functionality) and be able to send along with my request
>a large chunk of data, without storing that chunk into memory.
>
>To clarify my need here is an example
>
>(1) client -> POST <large data file> -> server
>(2) server computes
>(3) server -> <REPLY with large data file> -> client
>
>the (3) step is done in a memory efficient way using
>t.w.c.downloadPage(). Right now I execute the (1) step using the
>"postData" argument of t.w.c.downloadPage after loading all data into
>memory.
>
>Is there a way I can avoid loading all data into memory?

Twisted 9.0 introduced twisted.web.client.Agent which is intended to 
eventually replace getPage, downloadPage, and all the other stuff in the 
twisted.web.client module.

If you're lucky, then Agent already supports all the features you're 
interested in, depending on Twisted 9.0 is okay for you, and you can 
switch to using it right away.  There are some helpers still missing, so 
you'll find it a little clunkier to use than downloadPage, but it will 
let you send your upload without loading it all into memory.

Otherwise, you might have to move to using HTTPClientFactory or 
HTTPDownloader directly (the factories used by getPage and downloadPage) 
in order to feed it data incrementally.  Avoid this if you can, those 
two factories are a mess.

Jean-Paul