[Twisted-Python] newbie confusion - puzzling reactor response
glyph at twistedmatrix.com
Fri Feb 12 23:13:57 EST 2010
On Feb 12, 2010, at 9:51 PM, K. Richard Pixley wrote:
> I'm working with parallelized build servers. We often have raided
> disks, solid state disks, servers with huge amounts of disk cache
> specifically so that an entire build happens in memory, etc. File io is
> our bottleneck.
Yeah, this level of disk manipulation is past the point where a little bit less blocking on I/O will help... I imagine you've already got some kind of process/thread pooling solution already, or at least you'll need one.
> I think you probably are also forgetting about NFS. NFS isn't slower
> than native disk in terms of throughput, only in terms of latency, which
> is a fabulous opportunity for asyncronous file io. (Granted, NFS seems
> to have fallen out of fashion recently.)
No, I'm not forgetting about it: I'm just saying that once you've got an API that applications can start using that gives *some* performance benefit (non-blocking disk I/O at the expense of spinning up a few threads / processes behind the scenes), you can always optimize it for other use-cases later, without necessarily changing the API.
> I think twisted already has everything that's required. It could
> probably use a slightly more friendly interface paradigm so the user
> doesn't have to do his own os.open, but really, even that wouldn't save
It would allow us to do it more portably, I think. os.open()'s behavior can vary a lot depending on what you do with it.
> Reactor core was enough to sell me on twisted. That's probably all I'll
> even be using. And compared to writing my own, that's enough to be useful.
Great, glad to hear it!
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python