[Twisted-Python] New C++ implementation of Deferreds

[Tom Cocagne]

    The C++ library is completely independent of Python and Twisted but it
can be made to work with them. One of the initial uses of it, in fact, was
to run a Python interpreter (using Twisted) in a dedicated C++ thread and
use it to direct the actions of all the other threads in the process.

    The way I did it is this:

    1. Spawn a C++ thread to run the Python interpreter inside of. All use
of the Python C-API must be done from within this thread if you want to
avoid dealing with the Global Interpreter Lock (remember Python itself is a
single-threaded application). Other C++ threads needing to call a Python
C-API function can do so using a DeferredCommand object that wraps the call
they want to make. Hand this object to the Python thread for execution and
use the Deferred mechanism to get the answer back to the original
thread..... Basically, you turn the interpreter into one big Active Object
as per the standard design pattern.

    2. Use the Python C-API to import a Python module you wrote that will
initialize Twisted and create a pair of TCP sockets that are connected to
each other. Then use the C-API to get the OS-level file descriptor to one of
the sockets. The purpose of this is to provide a mechanism for other C++
threads to inform the Twisted reactor that there's something in the C++
world it needs to do. When one of those DeferredCommands in step one is
given to the Python thread, the last thing the delivering thread needs to do
is call a function that writes a single byte to the TCP socket. This will
cause the Twisted reactor in the Python thread to wake up from it's 'select'
call and dispatch a handler. This handler can then execute a C++ function
you registered with the C-API. Once back in the C++ world and in the context
of the Python thread,  you can dispatch any of the DeferredCommands sitting
in the thread's queue.

    I don't currently have a mechanism for tying a Twisted Deferred directly
to one of the C++ deferreds since I haven't yet found a use-case where it
makes sense to do so. It's probably possible though.

    As for using the C++ deferreds with boost's asio... I haven't used asio
yet but a quick read through the API makes it looks like an almost perfect
match for these deferreds. In fact, I'd like to eventually put this code up
for consideration by the boost community for eventually including something
like it in boost. I *highly* doubt they'd want to use the code I've written
but it might serve as a good proof-of-concept at least.

    Tom

On Fri, Jan 30, 2009 at 12:09 AM, V S P <toreason at fastmail.fm> wrote:

> Hi, this is very interesting
> ( am I currently testing out C++ web server (using boost asio) called
> pion
> to see if can suit my needs (it is a multi-threaded async IO generic
> server and boost asio uses the best async APIs for each OS (including
> kques for freebsd for example).
>
>
> I looked at the example threads.cpp -- but could not figure out -- how
> would it integrate with Twisted ?
>
> thanks
>
> On Thu, 29 Jan 2009 12:21:36 -0800, "Tom Cocagne"
> <tom.cocagne at gmail.com> said:
> >     The question of whether anyone had developed a C++ implementation of
> > Twisted's Deferreds has cropped up on this list several times over the
> > last
> > few years. To date, I've unfortunately been forced to sit on an
> > implementation I developed several years ago for the Department of
> > Defense.
> > Due to some rather draconian IP & security policies, it's taken an
> > exceptionally long time to get the code released. I'm happy to announce
> > though that I finally managed to obtain release authorization for it and
> > that the code now has a new home on Sourceforge.
> >
> >     As C++ has differing language features, the API of the C++ deferreds
> > isn't identical to those of Twisted's but most of the key concepts mapped
> > over pretty well. Additionally, this implementation has a few design
> > extensions to facilitate the development multi-threaded C++ applications.
> > The multi-threaded aspect grew almost accidentally out of the original
> > single-threaded design but, since then, it has proven to be
> > extraordinarily
> > useful for simplifying several multi-threaded applications I've worked
> > on.
> >
> >     The implementation of the library is, however, rather complex. In
> >     large
> > part, this is due to the effort required to work around the inherent
> > inflexibility of the C++ type system. The Function, Bind, and
> > Meta-Programming libraries found in Boost have gone a long way towards
> > simplify the implementation though (my original Loki-based attempt was
> > almost incomprehensible even to me ;-). The library is fully documented
> > though so as long as you're reasonably comfortable with C++ templates, it
> > should be understandable.
> >
> >     So far, the library has only been exposed to a handful of individuals
> > for review. I'd be interested to hear the comments anyone has about the
> > design and/or implementation approach. The code is available at:
> >
> >         http://deferred.svn.sourceforge.net/viewvc/deferred/
> >
> >     Cheers,
> >
> >     Tom
> --
>  V S P
>  toreason at fastmail.fm
>
> --
> http://www.fastmail.fm - Accessible with your email software
>                          or over the web
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090130/fe302cc7/attachment.htm