[Twisted-Python] Really Basic clarification on defers

[John Aherne]

A point I missed out on:
The adbapi module seems to be a good example of using deferreds and threads.
The adbapi module returns a deferred it has created, you add your callbacks
to it. It calls your callback when ready. It does seem like the examplar for
doing deferreds.

The db stuff will normally block so put it in a thread and use deferreds to
get the result or failure.

A point about the db calls is that they can be very intensive. If you need
to run some db calls every 30 secs or 60 secs and the db takes 50% or more
of the time to generate the results, you won't have much time to service any
incoming requests to see the data results. The remote connections will be
failing bigtime.

So then I suppose you should break the code into 2 programs. One that does
the db stuff, the other to handle the remote connections. The db code when
it has a result will then connect to the other program and pass across its
results. There may be better ways of doing this of course.

So as Jarrod points out, deferToThread is an easy way of solving blocking
code, but not always. It seems good for short blocks, but you do need the
bulk of the time available for handling connections.

If anyone sees howlers in this, please let me know.

Regards

John Aherne



On Wed, Aug 5, 2009 at 11:04 PM, John Aherne <johnaherne at rocs.co.uk> wrote:

>
>
> On Wed, Aug 5, 2009 at 4:17 PM, <exarkun at twistedmatrix.com> wrote:
>
>> On 09:33 am, johnaherne at rocs.co.uk wrote:
>> >On Wed, Aug 5, 2009 at 12:14 AM, Johann Borck
>> ><johann.borck at densedata.com>wrote:
>> >>
>> >[snip]
>> >
>> >Sendline is not blocking so as you say we can avoid the use of
>> >deferreds and
>> >continue to use sendline directly.
>>
>> LineReceiver.sendLine is not blocking, correct.  However, your statement
>> implies that if it were blocking, you could use Deferreds to address
>> this
>> problem.  This is incorrect.  Deferreds do not make blocking APIs into
>> non-blocking APIs.
>> >[snip]
>> >
>> >Our option 3 using defertothread does use sendline from the thread.
>> >Your
>> >response implies that is OK since you say defertothread is threadsafe.
>> >Did
>> >you really mean that.
>>
>> deferToThread is not thread-safe: you may only call it from the reactor
>> thread
>> (the thread in which you called reactor.run).
>>
>> Since deferToThread runs the function you pass to it in a non-reactor
>> thread,
>> you may not use any non-thread-safe Twisted APIs in the function you
>> pass to
>> it.
>> >
>> >Once again thanks for a very good response. That has cleared up a lot
>> >of
>> >confusion.
>> >
>> >I suppose it would help if there was a paragraph at the start of the
>> >twisted
>> >documentation detailing what you have just said. So when they start on
>> >deferreds you have some sort of context in which to interpret what is
>> >being
>> >said
>>
>> A significant effort is presently underway to improve the documentation
>> about
>> Deferreds.  Any specific feedback you have about it would be much
>> appreciated.
>> :)
>>
>> Jean-Paul
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
> Jean-Paul
>
> Thanks for the clarifications.
>
> I assume what I meant by blocking was I would have to put a function into
> deferToThread and add some callbacks to return the result. And not directly
> calling sendline from the thread. This is what Jarrod suggested in his
> reply. If I'm wrong, please say so.
>
> I have read the thread regarding deferreds with interest, but did not feel
> I knew enough to contribute.
>
> I do feel qualified to ask some very daft questions. Unfortunately, I don't
> see too many daft questions being asked in the list. I reckon if I need to
> know the answer to them then some other people probably do as well but don't
> put themselves forward. I see it as a way to document information that is
> difficult to come by. And I really do appreciate the very good answers I get
>
> However, after this little foray, I probably feel able to comment:--
>
> The concept of deferreds is very simple. Everyone understands the concept -
> even I do. The issue is how and why and where you should use them in
> twisted.
>
> Some Basic getting Started Points.
>
> 1. For simple network activity do not use deferreds. They are not
> necessary. You can get a lot done without deferreds. And you don't know how
> to use them yet. The reactor and the select will process the outgoing and
> incoming buffers without blocking. Anyone familiar with networking and
> select will already understand this. Anyone not familiar will not realise it
> and needs to be made aware of how the select works.
>
> 2. If you have blocking code - please define blocking :), then first think
> about  putting it into deferToThread with appropriate callbacks and return
> the deferred. As suggested by Jarrod in his response.
>
> 3. John Goerzen in his Apress book Python Network Fundamentals has a very
> simple chat server example. With a few comments for the uninitiated, this
> would be a good starter. Possibly I could ask for permission to include it
> in some twisted HOWTO documentation for beginners with suitable copyright
> recognition.
>
> 4. With these few points as starters, maybe more people will be encouraged
> to get started with twisted. And if you know you can ignore deferreds until
> later you will find twisted is very simple to use and get some good results
> with little effort.
>
> 5. The emphasis on how deferreds work probably ought to be counterbalanced
> by some insight into how and why and where you would use them. For example,
> if you have a text file of 10000 lines you need to read in and summarize,
> presumably you would run this with deferToThread(+other options) and get the
> result via the callback. If someone has a better example please let me know.
>
> 6. Blocking code is always put into a thread or like, and a deferred
> callback or errback used to return the result or failure of the blocking
> code from the thread. See jarrod's response above
>
>
> You asked :)
> I'm giving
>
> I may be completely off track in what I have said above. And I would not
> want anyone to fall upon this mail and think it represents the gospel truth.
>
> Thank you for the response..
>
> John Aherne
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090806/f56dad02/attachment.html>