[Twisted-Python] multiple workers

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Thu Sep 2 16:12:18 EDT 2010


On 05:26 pm, ruslan.usifov at gmail.com wrote:
>Why it is not supported?

"Why" it is not supported is that no one has decided to implement and 
support it.  If it's interesting behavior for you, then we would 
completely welcome you implementing it, and we'll even maintain the 
support for it once you've done that. :)

If you were asking about what specific implementation details cause it 
not to work now (more of a "how" question, sort of), then the answer to 
that probably varies from reactor to reactor, but it is all about how 
things end up being shared across the multiple processes created by 
fork.

I want behaviour like nginx http://nginx.org/, and
>misunderstand why i can't implemented it throw twisted. Its' so easy. 
>Every
>process have it's own set sockets, and they doesn't share  this sockets
>between each other. "OnConnect" event happens only once and which 
>process
>handle this event depend on operation system(select epoll, kqueue), in 
>my
>case this happens like round robin(FreeBSD 7.2-RELEASE-p8). Where here 
>is
>unsupported behaviour?

So, for example, epoll descriptors do survive fork().  However, kqueue 
descriptors don't.  So one necessary change for kqueue reactor to 
support this kind of behavior is to have the reactor somehow re- 
initialize itself after the fork.

Another problem is that certain resources are not simply duplicated by a 
fork().  A specific example is the one you brought up in your earlier 
post.  A unix socket only has one entity corresponding to it in the 
filesystem.  Twisted takes responsibility for cleaning these up, but 
after you fork(), there are two unix sockets and still only one 
filesystem entity.  This confuses one of the processes, since it 
believes it needs to delete the file.  Hardly rocket science to fix, but 
it's a specific case which needs to be handled.

And I'm sure you'll come across quite a few more specific cases which 
need to be handled.  This might get us back to the "why" a little - 
actually ensuring that everything will work properly when arbitrary 
forks are added is a major challenge.  I don't see any way to do it 
comprehensively, really.  That would leave you with a long, long 
adventure of fixing one little issue at a time for months or even years 
to come.  And each problem would only become evident after it bit you 
somehow.

That's probably why we have a ticket for an explicit file descriptor 
passing API, rather than a ticket for supporting arbitrary fork calls. 
The former is easier to test and be confident in than the latter.

Jean-Paul



More information about the Twisted-Python mailing list