[Twisted-web] t.w.client.downloadPage with large amount of postData
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Fri Oct 8 09:23:20 EDT 2010
On 5 Oct, 06:45 pm, seadog at sealabs.net wrote:
>Hello everyone,
>I'm writing a client that is using t.w.client.downloadPage function to
>download a large chunk of data into a file from a webserver. This is
>working fine.
>
>Now I want use the same function (or at least take advantage of the
>"save to file" functionality) and be able to send along with my request
>a large chunk of data, without storing that chunk into memory.
>
>To clarify my need here is an example
>
>(1) client -> POST <large data file> -> server
>(2) server computes
>(3) server -> <REPLY with large data file> -> client
>
>the (3) step is done in a memory efficient way using
>t.w.c.downloadPage(). Right now I execute the (1) step using the
>"postData" argument of t.w.c.downloadPage after loading all data into
>memory.
>
>Is there a way I can avoid loading all data into memory?
Twisted 9.0 introduced twisted.web.client.Agent which is intended to
eventually replace getPage, downloadPage, and all the other stuff in the
twisted.web.client module.
If you're lucky, then Agent already supports all the features you're
interested in, depending on Twisted 9.0 is okay for you, and you can
switch to using it right away. There are some helpers still missing, so
you'll find it a little clunkier to use than downloadPage, but it will
let you send your upload without loading it all into memory.
Otherwise, you might have to move to using HTTPClientFactory or
HTTPDownloader directly (the factories used by getPage and downloadPage)
in order to feed it data incrementally. Avoid this if you can, those
two factories are a mess.
Jean-Paul
More information about the Twisted-web
mailing list