[Twisted-Python] Perspective Broker with blocking, non-thread-safe C api?

Brian Granger ellisonbg.net at gmail.com
Wed Jul 11 11:25:38 EDT 2007


Things are a bit more complicated that just calling deferToThread.

The reason is that is you call a blocking C api in the thread, the
Python interpreter will not be able to switch threads until the
blocking C calls....unless the C code releases the Global Interpreter
Lock.  Thus you have two choices:

1)  Make sure that the C code releases the GIL and call it using
deferToThread.  This is probably the best solution.

2) If you can't release the GIL - if for instance the C code makes
calls to the Python C API, then you have to do the following.  You can
simply call the block C code in the main thread.  This will block the
Twisted event loop while the C code runs.  To mask this blocking
behavior, you will need to have a client that takes this into account.
 You basically need a way of making sure that while that code is
running no other client makes a call to the server running the C code.
 You also have to make sure that the client that initiated the call
doesn't make any further calls until the first is done.

The way we usually handle this is to make a separate process that
manages a queue of commands to be executed on the server.  Clients
then connect to this manager, which in turns submits them to the
actual PB server at the appropriate time.  This is a pain, but it
works extremely well.

Hope that helps.

Brian
On 7/11/07, Phil Mayers <p.mayers at imperial.ac.uk> wrote:
> On Wed, 2007-07-11 at 00:34 +0000, s. w. wrote:
> > I'd like to use Perspective Broker to serve a set of objects remotely.
> > However, these objects wrap a non-thread-safe, blocking C api.  I'm new to
> > Twisted, but I believe I understand correctly that the blocking C api needs
> > to be accessed via threads in Twisted.
> >
> > Given this, the only way I can think to use Perspective Broker would be to
> > use a special PBServerFactory/Protocol that limited the server to one
> > connected client.  For the use I would be putting the server to, this is
> > maybe not as horrible as it sounds, but it is probably as ugly as it sounds.
> >
> > I'm hoping there's something in PB that I haven't come across that can solve
> > this problem in a more elegant way.  Any help or pointers would be greatly
> > appreciated.
>
> It's not specific to PB - this applied to all of Twisted, and all of
> Twisted has the same solution.
>
> See "deferToThread" for examples of ways to call blocking APIs inside a
> thread, and when the call returns the deferred will fire.
>
> >
> > Thanks,
> > Steve
> >
> > _________________________________________________________________
> > http://newlivehotmail.com
> >
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>




More information about the Twisted-Python mailing list