[Twisted-web] Progress on client POST methods

[Jean-Paul Calderone]

On Sun, 07 Jan 2007 17:37:44 +0000, Ross Burton <ross at burtonini.com> wrote:
>Hi,
>
>I'm writing a Flickr upload tool[1] and would like to be able to show
>the progress of the image upload.  Images are uploaded via HTTP POST:
>
>from twisted.web import client
>...
>return client.getPage("http://api.flickr.com/services/upload/",
>                      method="POST",
>                      headers=headers,
>                      postdata=form)
>
>where form is the encoded title/tags/image data.  Is there a way to get
>progress information from twisted.web.client?  If I'm uploading 4M
>images the upload takes some time to complete, so I'd like to be able to
>have a progress bar.

Take a look at twisted.web.client.HTTPPageGetter's implementation.  Note
the last thing it does in connectionMade:

    if data is not None:
        self.transport.write(data)

If you want progress reporting, then this is the code to fix in order to
get it.  Rather than writing all of the data at once, this code could use
a producer to write it approximately only as quickly as the peer reads it.
This will let you generate a progress report, since you can keep track of
how many bytes you have written and how many bytes are yet to be written.

Due to the factoring of this code, I don't really see any way to provide
this feature without at least copying all of the code in the existing
connectionMade method and then only changing the last two lines, unless you
refactor it to be more friendly to this use case and contribute a patch.
Such a patch would be welcome, though, if you choose to do this.

Some producer documentation is available here:

http://twistedmatrix.com/projects/core/documentation/howto/producers.html

>
>Also, should I be using twisted.web2.client now, or is it still under
>development?

It is still under development (like much of Twisted - but more likely to
change incompatibly than much of Twisted).

Jean-Paul