[Twisted-web] Is there a way to implement a deferred sequence?

Alexey S. twisted-web at udmvt.ru
Fri Apr 10 04:12:28 EDT 2009

On Thu, Apr 09, 2009 at 09:18:39PM +0200, Terry Jones wrote:
> >>>>> "Alexey" == twisted-web  <twisted-web at udmvt.ru> writes:
> Hi Alexey
> Alexey> I want a sequence which is Deferred with two callbacks: one as
> Alexey> usual - for the whole result called at the end of the sequence,
> Alexey> another - called for every element of the sequence (as they
> Alexey> arrive).
> I'm pretty sure I don't understand exactly what you want. Maybe you could
> write it in pseudo-code, showing both how you'd call the thing you want and
> what the code to implement it would look like, at least conceptually. I
> can't promise to help, if you can be a bit more specific it might be easier
> to be more helpful.
> You might find something of worth in one of my Twisted mailing list
> soliloquies, which starts here:
>   http://www.twistedmatrix.com/pipermail/twisted-python/2008-June/017904.html
> I have feeling that the code Esteve Fernandez and I came up with to tackle
> that situation might be something like what you're after.
Well, that is close to my problem, but I have a feature request with somewhat more
generic requirements.

Current Deferred is used to defer a simple process, that only
have its 'start' and 'end', returning some single result object at the end.
And there is nothing between start and end, it is not allowed to look inside
that process. Once started it can only fail or finish returning a single result.

But what about deferring some more complex process, that may have zero to more
itermediate results before finishing? What if we are inside that process and have
some event, but we are not finished the process? Why can't the consumers of the
Deferred be interested in receiving that event too?
Why not augment the Deferred for that?

As an example consider running deferred the final state machine. Having only a current
Deferred it is not possible to trace the state change if the state is not changed
to the final one, since there is no callback that can be called several times for
every state change. We can only know, when the machine is entered the final state
and stopped or failed.

It is not possible to defer a process that run in stages if each stage does return some
result or if we want to trace the completion of stages, not only the whole process.
(Not possible without further complications, described in the link above.)

It would be nice if Deferred be extended to have another callback to be called for
updating the result or notifying about state change of some process.
In addition to 
we will have something like the

How this may be used depends on what the process is and what are the intermediate results.
For example d.update_state may be invoked to tell the percentage of completion of the whole deferred process.
Or d.update_state may be invoked to append new item to the resulting sequence as it arrives.
Or d.update_state may be invoked to say that intermediate result should be replaced by a new one
or somehow updated in other fashion.
That could also depend on the type of the final result:
   If we are returning a list, then we may want to append items to it.
   If we are returning a tree we may want to build that tree node by node. (may we want that?)
   If we are aggregating some data series into some single result we may want to present
    current intermediate result to the observer.
   Any other process, that could be useful to observe in its dynamics, not only in its final state.

It is just the way to send/receive the events produced by some process that is deferred
in the case that events are not the 'process_finished' events.

I think that Deferred class can be extended in that way preserving full backward compatibility,
since if noone is interested in recieving intermediate events, noone will register
a callback for that and no callback will be called - noone will notice that.

> You may also like to think about whether you can use
> twisted.internet.task.LoopingCall It's more flexible than it may appear at
> first, and it also fits your pattern at least at a high-level description.
Well, it can solve only a specific problem, as far as I understand.

> Terry

Thank for your attention and interest.

PS: Should I file a feature request? Or that feature is absolutely unnecessary to have?
PPS: Does anybody need some more detailed explanation?

Alexey S.

More information about the Twisted-web mailing list