[Twisted-Python] Deferred in C++
jkakar at kakar.ca
Wed Oct 22 05:05:43 EDT 2008
On Tue, Oct 21, 2008 at 5:14 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> So I am not the only one who tries to use Twisted concepts in C++.
Heh, it's lonely being a C++ nerd, isn't it. ;) I'm mostly
interested in seeing what's possible and trying to understand how to
port very interesting APIs from Python to C++.
> I did write some kind of a Deferred class in C++ myself, together with
> a re-implementation of a Twisted framework (two reactors featuring TCP
> client & server, callLater, callInThread...; a selectReactor and a
> native MFC reactor that merges nicely in a Windows application, yes,
> there are some; Protocols, LineReceiver, HttpClient & server...)
> and I already have a running business application that uses all of it
> on Windows.
> The Deferred part was really hard to implement, and may be the reason
> why it was never done before. It really pushed me out of my C++
> I tried to convert my code to use your class, which seems easier to
> use than mine. Great job!
The most difficult part for me was dealing with Deferred-returning
callbacks and getting the pausing/unpausing behaviour to work
correctly. The difficult part was that all the template functions
caused order of definition issues. The DeferredTypeInfo was
introduced for this case to make Deferred a template parameter and,
therefore, work around the ordering issues. It's really heinously
complicated inside and I have to decipher it each time I look at it.
It's always a bad sign when you're the author of the code you can't
> - First, good news, your code almost compiles on Windows (VS 2005). I
> had to typedef int32_t (why don't you simply use int?), comment out
> the StackFrameFactory, and declare some functions/classes with
> __declspec(dllexport) (with the help of a macro) so we can build a
Cool, glad to hear it. I didn't expect problems, but just haven't
had time to port it (and also have VS2005 as my base target compiler
for Windows). I plan to s/int32_t/int/, but haven't gotten around
to it. It's a historic artifact in that code from many years ago
that I've continued for the time being to keep things consistent.
> - In my eyes, the notation "addCallback" is more readable than
> "add_callback" (and it's the one used by twisted python), but that's a
> matter of taste. On the other hand I found disturbing at first to fire
> deferreds with the methods "succeed()" and "fail()" (instead of
> callback() and errback()
The style issue is entirely personal, so yeah, I can see why you
feel that way. As far as succeed() and fail() vs. callback() and
errback(), this change was a suggestion from Christopher Armstrong.
Apparently, a lot of Twisted users have been confused by callback
(the callable you add to a Deferred) vs callback (the method you
call to fire the Deferred), and similar for errback.
> - I found that my callbacks often have a void result, and deferred are
> sometimes fired without a value. For example, a twisted DeferredList
> often just waits for its children to finish. The results may be
> gathered somewhere, but the callbacks return None.
> I took the liberty to change your code and add *a lot* of template
> specializations for the "void" ReturnType or ResultType.
> I suggest that when a callback takes no parameter, it can be added to
> any Deferred, whatever its current state (and the current value is
> simply discarded).
This is on my list of things to add, but haven't gotten around to it
yet. My plan is to throw a NoResultError or something whenever a
callback is added to a Deferred that already has void-returning
callback at the end of the chain. Though, hmm... I guess you might
still want to add an errback to handle errors from the
> - In some cases my callbacks take additional arguments (the twisted
> "callbackArgs"). To make it possible I used the
> boost::bind functions, and a wrapper class that converts any
> boost::function to a ion::ICallback.
Ah, nice. Yeah, I was wondering about that and have been
considering implementing a specialized ICallback that can store
> - Several times I had a "Can't add invalid callback" error, and I
> found it difficult to figure which type was in the deferred state (the
> previous callback returned a Deferred, and so on...) So I extended
> ITypeInfo with a "const char* typename" member, that I fill with
> "typeid(Object).name()". Much easier to see with the debugger...
Good call. Yeah, I've had the same issue interpreting
> - I tried to implement a DeferredList. I almost succeeded, except that
> it simply inherit from Deferred and it seems a bad idea: when the
> DeferredList is returned as a plain Deferred, all state is lost... and
> the application crashes.
> Maybe Deferred should be a kind of "shared pointer" to the real
> object: DeferredState or DeferredListState
Deferred is a "shared pointer" to DeferredState...? Maybe I'm not
understanding your point?
> - Errr... I'm not sure I understood the "trule" concept. When do you
> use it? What are its semantics regarding ownership? is it similar to
This is an implementation of the Holder/Trule pattern described in
Josuttis' "C++ Templates" book. There is a reason, which I can't
recall right now, that makes returning a ptr (or auto_ptr) from a
function potentially dangerous (and also passing one in) that trule
My plan is to migrate to the Boost pointers that will become part of
the standard library in C++0x. This will make working with smart
pointers in ion much nicer. The ptr/trule pattern is quite
cumbersome, especially when templates are involved.
> - How is one supposed to get the content of a Failure? I use code like
> this, but it seems too long for this simple task:
> void MyProcess::displayError(ion::trule<ion::Failure> failure)
> std::string message = ion::ptr<ion::Failure>(failure)->get_message();
> throw ion::UnhandledFailureError(failure);
> - likewise, it should be possible to simply
> throw Failure("Some error message");
The displayError above is what you're expected to do, so far. If
there's a better way I'd love to hear it. There's actually a deeper
issue that makes Failure/error handling in this implementation kind
of crappy. The type of an exception is lost when caught in a catch
block for a base class of the thrown exception. Which means that
it's not possible, currently, to determine the concrete exception
type from a Failure. One thing I've been considering is making the
ion::Exception class provide some extra functionality to make this
possible, but I'm unhappy that doing that will make implementing
custom ion::Exception's harder.
Being able to throw a Failure with a string message is a good idea.
>> The reliance on templates would negatively affect
>> compile time, for one thing. More importantly, I've found
>> deciphering the error messages produced by simple mistakes can be
> I confirm: it is.
>> Nonetheless, my main motivation was to see if it was
>> possible at all and so I'm pleased that it works in the end.
> I think my program can be considered as a non-trivial application (I
> cannot show its code, though, except for the non-business framework)
> It is definitely possible to implement a Twisted framework in C++. The
> hard part is to make it nice for the developer. Programming with
> callbacks already twists your mind, it becomes very difficult if the
> language syntax hides the important part of the code.
This "make it nice for the developer" is the primary goal I have
with this effort.
> The "ion" project seems promising! What extend do you intend to give it?
I'm interested in getting to the point where there's a Twisted-style
reactor and, eventually, support for using AMP-based protocols.
More information about the Twisted-Python