[Twisted-Python] deciding to use twisted or not

[Glyph Lefkowitz]

On Wed, Aug 26, 2009 at 4:56 PM, Martin-Louis Bright <mlbright at gmail.com>wrote:

> I would like to write a small daemon that monitors (tails) a server log,
> parses the entries and sends HTTP requests based on some of those entries. I
> would like it if the reading of the log file and the sending of http
> requests were asynchronous. Should I use twisted for this? Or is twisted
> overkill...
>
Twisted would be perfectly appropriate for this!  The HTTP requests will
certainly be non-blocking, assuming you use twisted.web.client (or the new,
better thing which I hope will be released before we all get old).

In fact, if anything, Twisted is under-kill; you'll need to go a bit
further.  The one minor issue is that Twisted has no explicit way to do
asynchronous file I/O (because most operating systems provide a bewildering
array of not-really-working systems for non-blocking file I/O, so it would
be challenging for Twisted to do in a way that was useful).  There are a
number of ways to emulate it though; for log files (which are fairly low
volume, and almost by definition will immediately be in your filesystem
cache as the parts you want to read become available) you can just do a
LoopingCall() which does .read() on a file that it leaves open to retrieve
the next chunk of data, and that will probably be good enough.  In practice
you won't actually block doing this read() because your kernel is just going
to pull the bytes directly out of filesystem cache memory and hand them back
to you.  It is of course possible to have to wait for the disk or even the
network, depending on your underlying filesystem.

I just filed http://twistedmatrix.com/trac/ticket/3983 for a more thorough
solution (mostly in the hopes that somebody will close it as a duplicate and
point to some pre-existing issue I couldn't find through search) so you can
monitor future discussion there if you like.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090826/076162d0/attachment.html>