[Twisted-Python] Really Basic clarification on defers

[John Aherne]

On Wed, Aug 5, 2009 at 4:17 PM, <exarkun at twistedmatrix.com> wrote:

> On 09:33 am, johnaherne at rocs.co.uk wrote:
> >On Wed, Aug 5, 2009 at 12:14 AM, Johann Borck
> ><johann.borck at densedata.com>wrote:
> >>
> >[snip]
> >
> >Sendline is not blocking so as you say we can avoid the use of
> >deferreds and
> >continue to use sendline directly.
>
> LineReceiver.sendLine is not blocking, correct.  However, your statement
> implies that if it were blocking, you could use Deferreds to address
> this
> problem.  This is incorrect.  Deferreds do not make blocking APIs into
> non-blocking APIs.
> >[snip]
> >
> >Our option 3 using defertothread does use sendline from the thread.
> >Your
> >response implies that is OK since you say defertothread is threadsafe.
> >Did
> >you really mean that.
>
> deferToThread is not thread-safe: you may only call it from the reactor
> thread
> (the thread in which you called reactor.run).
>
> Since deferToThread runs the function you pass to it in a non-reactor
> thread,
> you may not use any non-thread-safe Twisted APIs in the function you
> pass to
> it.
> >
> >Once again thanks for a very good response. That has cleared up a lot
> >of
> >confusion.
> >
> >I suppose it would help if there was a paragraph at the start of the
> >twisted
> >documentation detailing what you have just said. So when they start on
> >deferreds you have some sort of context in which to interpret what is
> >being
> >said
>
> A significant effort is presently underway to improve the documentation
> about
> Deferreds.  Any specific feedback you have about it would be much
> appreciated.
> :)
>
> Jean-Paul
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

Jean-Paul

Thanks for the clarifications.

I assume what I meant by blocking was I would have to put a function into
deferToThread and add some callbacks to return the result. And not directly
calling sendline from the thread. This is what Jarrod suggested in his
reply. If I'm wrong, please say so.

I have read the thread regarding deferreds with interest, but did not feel I
knew enough to contribute.

I do feel qualified to ask some very daft questions. Unfortunately, I don't
see too many daft questions being asked in the list. I reckon if I need to
know the answer to them then some other people probably do as well but don't
put themselves forward. I see it as a way to document information that is
difficult to come by. And I really do appreciate the very good answers I get

However, after this little foray, I probably feel able to comment:--

The concept of deferreds is very simple. Everyone understands the concept -
even I do. The issue is how and why and where you should use them in
twisted.

Some Basic getting Started Points.

1. For simple network activity do not use deferreds. They are not necessary.
You can get a lot done without deferreds. And you don't know how to use them
yet. The reactor and the select will process the outgoing and incoming
buffers without blocking. Anyone familiar with networking and select will
already understand this. Anyone not familiar will not realise it and needs
to be made aware of how the select works.

2. If you have blocking code - please define blocking :), then first think
about  putting it into deferToThread with appropriate callbacks and return
the deferred. As suggested by Jarrod in his response.

3. John Goerzen in his Apress book Python Network Fundamentals has a very
simple chat server example. With a few comments for the uninitiated, this
would be a good starter. Possibly I could ask for permission to include it
in some twisted HOWTO documentation for beginners with suitable copyright
recognition.

4. With these few points as starters, maybe more people will be encouraged
to get started with twisted. And if you know you can ignore deferreds until
later you will find twisted is very simple to use and get some good results
with little effort.

5. The emphasis on how deferreds work probably ought to be counterbalanced
by some insight into how and why and where you would use them. For example,
if you have a text file of 10000 lines you need to read in and summarize,
presumably you would run this with deferToThread(+other options) and get the
result via the callback. If someone has a better example please let me know.

6. Blocking code is always put into a thread or like, and a deferred
callback or errback used to return the result or failure of the blocking
code from the thread. See jarrod's response above


You asked :)
I'm giving

I may be completely off track in what I have said above. And I would not
want anyone to fall upon this mail and think it represents the gospel truth.

Thank you for the response..

John Aherne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090805/754a8c84/attachment.html>