[Twisted-Python] Really Basic clarification on defers

[John Aherne]

On Wed, Aug 5, 2009 at 12:14 AM, Johann Borck <johann.borck at densedata.com>wrote:

>
> On Tue, Aug 4, 2009 at 10:08 AM, John Aherne <johnaherne at rocs.co.uk
> <mailto:johnaherne at rocs.co.uk>> wrote:
> > This is a really basic problem we are trying to decide about,
> >
> > We have programs that run quite happily, so far. Its main task is to
> > receive data from port A and send it out via port B. Then receive data
> > via port B and send it out via port A. It's pretty much like a chat
> > setup. You just build up a list of connected clients and send data to
> > them as required
> >
> > One side A receives some input from a tcp port - about 100-200
> > characters, and forwards it to another port B. We do not need to wait
> > for any response. If we get a response we pick that up through line
> > receiver. We also run a calllater to check if we got a response on
> > linereceiver within the timeframe specified. If not we drop the
> > connection.
> >
> > Traffic coming in from port B is analysed and some subset is sent back
> > to port A.
> >
> > Ignoring port A for the moment, just concentrating on port B, we have
> > tried three options:--
> >
> > 1. We set up a defer to handle the sendline to port B so that the
> > reactor would schedule it in its own good time.
>
> The reactor always schedules reads and writes "in its own good time",
> which means it writes whenever there's data to write and the socket is
> ready for writing. If you have data that can't be written at once,
> because it's too much for the socket to handle in a non-blocking
> fashion, the reactor (along with the transport) will take care of it,
> and defer its delivery itself, no need for any deferreds you'd had to
> care about here.
>
> Correct me if I'm wrong, but as I understand your description, option 1.
> and 2. do not behave identically. This is how I interpret it:
> option 1:
>
> A sends msg1 to [svc] : wrap msg1 in deferred1
> [ - time - ]
> B sends data? to [svc] :
>                                        1. callback deferred1: [svc]
> sends msg1 to B
>                                        2. handle data?
> B sends rsp1 to [svc]: [svc] sends rsp1 to A
>
> option 2:
>
> A sends msg1 to [svc] : [svc] sends msg1 to B
> B sends rsp1 to [svc] :  [svc] sends rsp1 to A
>
> If this is the case, you rely on some data? being sent to [svc] before
> msg1 can be forwarded to B. That means that you have msg1 in memory
> until you receive data? from B. This doesn't cause problems in your
> case, since you handle small messages in big intervals. But if you'd
> increase the load significantly, you'd also need significantly more RAM
> for no good reason. A case where option 1 might make sense would be if
> it depended on data? provided by B, to decide if or how to continue
> processing msg1. Then you had a valid use-case for deferreds. Since
> there are no such requirements, option 2 is definitely the right choice.
>
> > No threads involved using the standard twisted setup. When we get a
> > response through receiveline we fire the callback defer. If we timeout
> > via callLater we fire the errback to clear the defer. In this case the
> > defer does not seem to be doing very much
> >
> > 2. Now a fresh pair of eyes is looking at the code and saying why are
> > we using a deferred for sending data to port B. We could just issue a
> > straight sendline as part of the main code and carry on. If we get a
> > response via linereceiver,we process it normally, otherwise we set our
> > callLater running and timeout and lose the connection. So no deferreds
> > required at all. It does seem to work.What we are not sure about is
> > what penalty is incurred in terms of reliability or throughput by
> > using sendline without a deferred.
>
> There's absolutely no penalty (unless you allow the notion of negative
> penalties). Using sendline directly is faster than using a deferred in
> between, even if you don't count the memory overhead. I think there's a
> bit confusion about the role of deferreds in twisted here. Deferreds
> don't help you (or the reactor) with scheduling, they only provide you
> with a means to continue some processing after a certain event occurred.
> > We are not too sure what the holdup will be and whether it could end
> > up halting the show. Is it better to schedule these messages via
> > deferreds or am I missing something obvious
> >
> > 3. So we then did an experiment and used defertothread to run the
> > sendline in a separate thread with its own defer to maximise the
> > asynchronous running of the code. So now we are running threads when
> > one of the reasons for looking at twisted was that we could avoid
> > threads as much as possible.
>
> Do you use sendline (the twisted api) from within the thread?  If yes
> and it works, it works accidentally, probably also due to the very small
> load, and is definitely wrong (as well as unnecessary), twisted is not
> threadsafe, with the exception of a few methods/functions like
> callInThread/callFromThread/defertoThread etc.
>
>
> hope that helps,
> Johann
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>


Johann

Thanks for a very informative reply.

Our problem was more basic than deferreds. It was to with the reactor and
select command.

Sendline is not blocking so as you say we can avoid the use of deferreds and
continue to use sendline directly.

Regarding messages being stored in memory, we only store the sequence number
for each message so we can identify the message being responded to. We can
then report back to A with the original message sequence number. So the
memory footprint is quite small.

Our option 3 using defertothread does use sendline from the thread. Your
response implies that is OK since you say defertothread is threadsafe. Did
you really mean that.

Once again thanks for a very good response. That has cleared up a lot of
confusion.

I suppose it would help if there was a paragraph at the start of the twisted
documentation detailing what you have just said. So when they start on
deferreds you have some sort of context in which to interpret what is being
said

Regards

John Aherne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090805/14ee349e/attachment.htm