[Twisted-Python] newbie confusion - puzzling reactor response

[Glyph Lefkowitz]

On Feb 12, 2010, at 9:51 PM, K. Richard Pixley wrote:

> I'm working with parallelized build servers.  We often have raided 
> disks, solid state disks, servers with huge amounts of disk cache 
> specifically so that an entire build happens in memory, etc.  File io is 
> our bottleneck.

Yeah, this level of disk manipulation is past the point where a little bit less blocking on I/O will help... I imagine you've already got some kind of process/thread pooling solution already, or at least you'll need one.

> I think you probably are also forgetting about NFS.  NFS isn't slower 
> than native disk in terms of throughput, only in terms of latency, which 
> is a fabulous opportunity for asyncronous file io.  (Granted, NFS seems 
> to have fallen out of fashion recently.)

No, I'm not forgetting about it: I'm just saying that once you've got an API that applications can start using that gives *some* performance benefit (non-blocking disk I/O at the expense of spinning up a few threads / processes behind the scenes), you can always optimize it for other use-cases later, without necessarily changing the API.

> I think twisted already has everything that's required.  It could 
> probably use a slightly more friendly interface paradigm so the user 
> doesn't have to do his own os.open, but really, even that wouldn't save 
> much.

It would allow us to do it more portably, I think.  os.open()'s behavior can vary a lot depending on what you do with it.

> Reactor core was enough to sell me on twisted.  That's probably all I'll 
> even be using.  And compared to writing my own, that's enough to be useful.

Great, glad to hear it!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100212/2dcf9499/attachment.html>