[Twisted-Python] Deferred in C++

Jamu Kakar jkakar at kakar.ca
Wed Oct 22 05:05:43 EDT 2008

Hi Amaury,

On Tue, Oct 21, 2008 at 5:14 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> So I am not the only one who tries to use Twisted concepts in C++.

Heh, it's lonely being a C++ nerd, isn't it. ;) I'm mostly
interested in seeing what's possible and trying to understand how to
port very interesting APIs from Python to C++.

> I did write some kind of a Deferred class in C++ myself, together with
> a re-implementation of a Twisted framework (two reactors featuring TCP
> client & server, callLater, callInThread...; a selectReactor and a
> native MFC reactor that merges nicely in a Windows application, yes,
> there are some; Protocols, LineReceiver, HttpClient & server...)
> and I already have a running business application that uses all of it
> on Windows.
> The Deferred part was really hard to implement, and may be the reason
> why it was never done before. It really pushed me out of my C++
> knowledge.
> I tried to convert my code to use your class, which seems easier to
> use than mine. Great job!

The most difficult part for me was dealing with Deferred-returning
callbacks and getting the pausing/unpausing behaviour to work
correctly.  The difficult part was that all the template functions
caused order of definition issues.  The DeferredTypeInfo was
introduced for this case to make Deferred a template parameter and,
therefore, work around the ordering issues.  It's really heinously
complicated inside and I have to decipher it each time I look at it.
It's always a bad sign when you're the author of the code you can't
comprehend. :)

> - First, good news, your code almost compiles on Windows (VS 2005). I
> had to typedef int32_t (why don't you simply use int?), comment out
> the StackFrameFactory, and declare some functions/classes with
> __declspec(dllexport) (with the help of a macro) so we can build a
> DLL.

Cool, glad to hear it.  I didn't expect problems, but just haven't
had time to port it (and also have VS2005 as my base target compiler
for Windows).  I plan to s/int32_t/int/, but haven't gotten around
to it.  It's a historic artifact in that code from many years ago
that I've continued for the time being to keep things consistent.

> - In my eyes, the notation "addCallback" is more readable than
> "add_callback" (and it's the one used by twisted python), but that's a
> matter of taste. On the other hand I found disturbing at first to fire
> deferreds with the methods "succeed()" and "fail()" (instead of
> callback() and errback()

The style issue is entirely personal, so yeah, I can see why you
feel that way.  As far as succeed() and fail() vs. callback() and
errback(), this change was a suggestion from Christopher Armstrong.
Apparently, a lot of Twisted users have been confused by callback
(the callable you add to a Deferred) vs callback (the method you
call to fire the Deferred), and similar for errback.

> - I found that my callbacks often have a void result, and deferred are
> sometimes fired without a value. For example, a twisted DeferredList
> often just waits for its children to finish. The results may be
> gathered somewhere, but the callbacks return None.
> I took the liberty to change your code and add *a lot* of template
> specializations for the "void" ReturnType or ResultType.
> I suggest that when a callback takes no parameter, it can be added to
> any Deferred, whatever its current state (and the current value is
> simply discarded).

This is on my list of things to add, but haven't gotten around to it
yet.  My plan is to throw a NoResultError or something whenever a
callback is added to a Deferred that already has void-returning
callback at the end of the chain.  Though, hmm... I guess you might
still want to add an errback to handle errors from the
void-returning callback.

> - In some cases my callbacks take additional arguments (the twisted
> "callbackArgs"). To make it possible I used the
> boost::bind functions, and a wrapper class that converts any
> boost::function to a ion::ICallback.

Ah, nice.  Yeah, I was wondering about that and have been
considering implementing a specialized ICallback that can store

> - Several times I had a "Can't add invalid callback" error, and I
> found it difficult to figure which type was in the deferred state (the
> previous callback returned a Deferred, and so on...) So I extended
> ITypeInfo with a "const char* typename" member, that I fill with
> "typeid(Object).name()". Much easier to see with the debugger...

Good call.  Yeah, I've had the same issue interpreting

> - I tried to implement a DeferredList. I almost succeeded, except that
> it simply inherit from Deferred and it seems a bad idea: when the
> DeferredList is returned as a plain Deferred, all state is lost... and
> the application crashes.
> Maybe Deferred should be a kind of "shared pointer" to the real
> object: DeferredState or DeferredListState

Deferred is a "shared pointer" to DeferredState...?  Maybe I'm not
understanding your point?

> - Errr... I'm not sure I understood the "trule" concept. When do you
> use it? What are its semantics regarding ownership? is it similar to
> std::auto_ptr?

This is an implementation of the Holder/Trule pattern described in
Josuttis' "C++ Templates" book.  There is a reason, which I can't
recall right now, that makes returning a ptr (or auto_ptr) from a
function potentially dangerous (and also passing one in) that trule

My plan is to migrate to the Boost pointers that will become part of
the standard library in C++0x.  This will make working with smart
pointers in ion much nicer.  The ptr/trule pattern is quite
cumbersome, especially when templates are involved.

> - How is one supposed to get the content of a Failure? I use code like
> this, but it seems too long for this simple task:
>  void MyProcess::displayError(ion::trule<ion::Failure> failure)
>  {
>    std::string message = ion::ptr<ion::Failure>(failure)->get_message();
>    [...]
>    throw ion::UnhandledFailureError(failure);
>  }
> - likewise, it should be possible to simply
>    throw Failure("Some error message");

The displayError above is what you're expected to do, so far.  If
there's a better way I'd love to hear it.  There's actually a deeper
issue that makes Failure/error handling in this implementation kind
of crappy.  The type of an exception is lost when caught in a catch
block for a base class of the thrown exception.  Which means that
it's not possible, currently, to determine the concrete exception
type from a Failure.  One thing I've been considering is making the
ion::Exception class provide some extra functionality to make this
possible, but I'm unhappy that doing that will make implementing
custom ion::Exception's harder.

Being able to throw a Failure with a string message is a good idea.

>> The reliance on templates would negatively affect
>> compile time, for one thing.  More importantly, I've found
>> deciphering the error messages produced by simple mistakes can be
>> tricky.
> I confirm: it is.

Heh. :)

>> Nonetheless, my main motivation was to see if it was
>> possible at all and so I'm pleased that it works in the end.
> I think my program can be considered as a non-trivial application (I
> cannot show its code, though, except for the non-business framework)
> It is definitely possible to implement a Twisted framework in C++. The
> hard part is to make it nice for the developer. Programming with
> callbacks already twists your mind, it becomes very difficult if the
> language syntax hides the important part of the code.

This "make it nice for the developer" is the primary goal I have
with this effort.

> The "ion" project seems promising! What extend do you intend to give it?

I'm interested in getting to the point where there's a Twisted-style
reactor and, eventually, support for using AMP-based protocols.


More information about the Twisted-Python mailing list