[Twisted-Python] Large file transfers

[Glyph Lefkowitz]

On Mon, 29 Jul 2002 12:35:31 +0300, Tommi Virtanen <tv at twistedmatrix.com> wrote:
> On Fri, Jul 26, 2002 at 07:41:27AM -0500, Chris Armstrong wrote:
> > AFAIK, twisted.web.static _does_ do non-blocking reading, so check
> > that out.  I don't think anyone has implemented non-blocking writing
> > yet. (There's some sort of FileWrapper protocol or somesuch IIRC,
> > but it just assumes that write() won't take very long -- it's only
> > for testing)

> 	Bzzt, wrong. All file IO is sync. To do async file IO,
> 	you need to use a different kernel API -- called AIO,
> 	and not very widely available. (Or emulate it by forking
> 	worker processes that pass data between stdin/stdout and
> 	file).

Mr. Virtanen is right. The reason you might think it's async is that most File
I/O operations in Twisted are performed in a response to some request (web,
ftp), and are treated as "large file" operations.  The file is read one 'chunk'
at a time, and more is only read when the network connection it's being written
to becomes available for writing again.  This is what the "producer/consumer"
API in twisted.internet is for.  In general, on linux, this will mean that your
data is available in most cases anyway, due to the speed difference between
network & file I/O.  In the cases where it's not, it's likely that your server
is under high enough load that it's OK to block for an ms or two to wait for
the data.

Asyncore has a great little comment about this:

# After a little research (reading man pages on various unixen, and
# digging through the linux kernel), I've determined that select()
# isn't meant for doing doing asynchronous file i/o.
# Heartening, though - reading linux/mm/filemap.c shows that linux
# supports asynchronous read-ahead.  So _MOST_ of the time, the data
# will be sitting in memory for us already when we go to read it.

Hope this is enlightening.

-- 
 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020729/137d0477/attachment.pgp