[Twisted-Python] Really Basic clarification on defers

[John Aherne]

This is a really basic problem we are trying to decide about,

We have programs that run quite happily, so far. Its main task is to receive
data from port A and send it out via port B. Then receive data via port B
and send it out via port A. It's pretty much like a chat setup. You just
build up a list of connected clients and send data to them as required

One side A receives some input from a tcp port - about 100-200 characters,
and forwards it to another port B. We do not need to wait for any response.
If we get a response we pick that up through line receiver. We also run a
calllater to check if we got a response on linereceiver within the timeframe
specified. If not we drop the connection.

Traffic coming in from port B is analysed and some subset is sent back to
port A.

Ignoring port A for the moment, just concentrating on port B, we have tried
three options:--

1. We set up a defer to handle the sendline to port B so that the reactor
would schedule it in its own good time. No threads involved using the
standard twisted setup. When we get a response through receiveline we fire
the callback defer. If we timeout via callLater we fire the errback to clear
the defer. In this case the defer does not seem to be doing very much

2. Now a fresh pair of eyes is looking at the code and saying why are we
using a deferred for sending data to port B. We could just issue a straight
sendline as part of the main code and carry on. If we get a response via
linereceiver,we process it normally, otherwise we set our callLater running
and timeout and lose the connection. So no deferreds required at all. It
does seem to work.What we are not sure about is what penalty is incurred in
terms of reliability or throughput by using sendline without a deferred. We
are not too sure what the holdup will be and whether it could end up halting
the show. Is it better to schedule these messages via deferreds or am I
missing something obvious

3. So we then did an experiment and used defertothread to run the sendline
in a separate thread with its own defer to maximise the asynchronous running
of the code. So now we are running threads when one of the reasons for
looking at twisted was that we could avoid threads as much as possible.

The conundrum we are trying to resolve now is which option should we use. Do
any of the options have a built-in problem awaiting the unwary. In theory
all 3 options work. But if No 1 works well enough for our volume of traffic
should we adopt that one. Or is it better to start using the defertothread
option. Is there a simple answer

The traffic is not large, upto a 100-200 remote devices on port B. They will
send GPS data every 20 secs, and about 500 messages of about 200 bytes
average throught the day. The remote devices will respond in an irregular
manner without dropping the connection, so we force a disconnectf if
important messages are not getting through. They are then forced to
reconnect.

We have looked through the code searching for enlightment and it does seem
to be well documented, but the information we are looking for comes well
before the doc strings.

Hopefully, someone can give us some pointers in the right direction.

Thanks for any help.

John Aherne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090804/da9d3c85/attachment-0001.htm