Opened 13 years ago

Closed 3 years ago

Last modified 3 years ago

#1009 defect closed fixed (fixed)

Document inlineCallbacks

Reported by: hypatia Owned by: Glyph
Priority: high Milestone:
Component: core Keywords: documentation
Cc: radix, Jean-Paul Calderone Branch: branches/inlineCallbacks-howto-1009-2
branch-diff, diff-cov, branch-cov, buildbot
Author: tomprince, glyph

Description


Attachments (2)

comments.txt (1.3 KB) - added by hypatia 12 years ago.
Alex Levy's examples with his comments.
defgen.txt (9.5 KB) - added by edsuom 11 years ago.

Download all attachments as: .zip

Change History (30)

comment:1 Changed 13 years ago by hypatia

http://mesozoic.geecs.org/cogito/archives/000160.html suggests that defgen
really needs a prominent place in the Deferred documentation.

There's one open question: how prominent a place in the documentation? When
should people be encouraged to use defgen, and when should they be encouraged to
use a more traditional callback setup?

comment:2 Changed 13 years ago by radix

As the author of defgen, I have a superior understanding of the subtleties 
involved in deciding when to use it.

That is to say, I don't know.

In my current work project, we haven't used it at all. I'm not really sure why. 
We didn't have many places in the project where we dealt with many Deferreds in 
a single method that we wanted serialized. I'm not sure if that's the end of it, 
but that's usually when I think to use defgen. However, thinking back, we 
probably could  have slightly improved the look of the code where we have 2 or 3 
callbacks attached to a Deferred. It would have at least moved those all into a 
single method.

I'm not sure if the stuff I'm saying here is useful at all, maybe it's even 
misleading. I'll try to have a sit down and think about this sometime soon.

As an aside, for my next work project, I'm looking forward to stabilizing trying 
out gthreadless in production....

comment:3 Changed 12 years ago by radix

Just a note.

scratch that about gthreadless. However, also note that we did eventually use
defgen quite a bit in our work projects. It turned out pretty well, but error
reporting still sucks a lot in certain cases.

Changed 12 years ago by hypatia

Attachment: comments.txt added

Alex Levy's examples with his comments.

comment:4 Changed 12 years ago by hypatia

Cc: hypatia removed
Component: conch
Owner: changed from hypatia to edsuom

comment:5 Changed 11 years ago by edsuom

Status: newassigned

I like defgen and have a use case for it, in a BNF terminal-evaluator project I'm working on:

def evaluator(*inputStates):
    def function():
        for inputState in inputStates:
            wfd = waitForDeferred(self.run(sentence, state))
            yield wfd
            resultState = wfd.getResult()
            results.append(f(inputState, resultState))
    function = deferredGenerator(function)

    return function()

I'll include this plus Levy's more advanced example in the docs.

comment:6 Changed 11 years ago by edsuom

Component: conchcore
Resolution: fixed
Status: assignedclosed

I wrote a tutorial document that is attached and also available as the "Deferred Generators" section of my "WinDictator Internals" document.

I suggest just pasting the source right into a wiki page of the Twisted site, or using wget to grab the HTML and put it in the regular docs, or both.

Changed 11 years ago by edsuom

Attachment: defgen.txt added

comment:7 Changed 11 years ago by edsuom

Keywords: review added

comment:8 Changed 9 years ago by Jean-Paul Calderone

Resolution: fixed
Status: closedreopened

comment:9 Changed 9 years ago by Jean-Paul Calderone

Cc: Jean-Paul Calderone added
Keywords: review removed
Owner: changed from edsuom to Thijs Triemstra
Status: reopenednew

This should be a howto and it should cover inlineCallbacks in addition to deferredGenerator and it should use some simple Deferred API like getPage.

comment:10 Changed 9 years ago by Jean-Paul Calderone

#1821 was a duplicate of this. There's a bunch of stuff attached to that ticket which should almost certainly be incorporated into a resolution to this ticket.

comment:11 Changed 7 years ago by <automation>

Owner: Thijs Triemstra deleted

comment:12 Changed 6 years ago by Jean-Paul Calderone

#5177 was a duplicate of this.

comment:13 Changed 6 years ago by Jean-Paul Calderone

Summary: Document defgenDocument inlineCallbacks and deferredGenerator

comment:14 Changed 6 years ago by Leonardo Santagada

From the duplicate bug because I think it is very important:

"There should be a way to show inlineCallbacks to new users to show that you can do asynchronous communication without using callbacks everywhere. In the last pycon(2011) guido himself told that he doesn't understand callback based code and that he thinks the future of async in python is using generators, so I really think this is something the twisted community should talk about."

One thing we may do is ask one of the bloggers that posted about this to lend the code or to rewrite it as a howto, but still in the rest of the docs there should be pointers to inlineCallbacks, not just a separate doc about it.

comment:15 Changed 5 years ago by Jean-Paul Calderone

Summary: Document inlineCallbacks and deferredGeneratorDocument inlineCallbacks

Don't bother documenting deferredGenerator, instead we'll deprecate it (#6044).

comment:16 Changed 5 years ago by Tom Prince

Author: tomprince
Branch: branches/inlineCallbacks-howto-1009

(In [37454]) Branching to inlineCallbacks-howto-1009.

comment:17 Changed 5 years ago by Tom Prince

(In [37455]) Apply deferredgenerator-howto-1821.patch from thijs.

Refs #1009, #1821.

comment:18 Changed 5 years ago by Tom Prince

I've applied the patch from #1821 (which was a lore translation of defgen.txt) and applied a minimal first pass at switching to documenting inlineCallbacks instead.

Review comments from #1821:

  • The use of the word intuitive in the introduction definitely has to go.
  • Using reactor.callLater is often confusing to people new to Deferreds. There should be some other event source here - eg, twisted.web.client.getPage
  • I'm not sure about the use of decorator syntax here. It should probably be replaced with manual decoration, although it may make sense to include at least one example using decorator syntax to help people who can't quite make the connection on their own. The Twisted coding standard forbids decorator syntax.
  • Only one of the code samples in this document can actually be run usefully. Runnable code is worth a lot more than dead examples.
  • The document needs to introduce the concepts which it will cover, explain what readers need to already be familiar with, and should include links to other related documents at the end.

comment:19 Changed 3 years ago by Glyph

Author: tomprincetomprince, glyph
Branch: branches/inlineCallbacks-howto-1009branches/inlineCallbacks-howto-1009-2

(In [43007]) Branching to inlineCallbacks-howto-1009-2.

comment:20 Changed 3 years ago by Glyph

Keywords: review added; core removed
Owner: set to dstufft

comment:21 Changed 3 years ago by Glyph

Completely rewritten prose documentation, in the introductory docs, featuring a lot less verbiage. (Also in ReST now because we already had lore's funeral)

comment:22 Changed 3 years ago by dstufft

Question, are there gotchas or problems from using inlineCallbacks that someone should be aware of?

comment:23 in reply to:  22 Changed 3 years ago by Glyph

Replying to dstufft:

Question, are there gotchas or problems from using inlineCallbacks that someone should be aware of?

You mean aside from "Concurrency is incredibly confusing and human brains can't really reason about it effectively?"

No, not really.

I wanted to include something about how yield suspends execution and so you have to worry about mutual exclusion and critical sections, but that's a very complicated topic to address in introductory documentation.

comment:24 Changed 3 years ago by Glyph

We could potentially have a separate ticket for the Deferred reference docs, but this ticket is like "inlineCallbacks should be documented at all".

comment:25 Changed 3 years ago by dstufft

Yea, that sounds more like something that should be in an advanced topic documentation linked from the introduction as a "more info" kind of thing. I've only done a quick once over, I'll go over it in more depth later this evening.

comment:26 Changed 3 years ago by dstufft

Keywords: review removed
Owner: changed from dstufft to Glyph

Ok coming back to this.

I think it looks good, my only suggestion is to add something like:

"Although inlineCallbacks looks similar to a synchronously written function, unlikely a synchronous function wherever a yield is used execution of the function will be paused while waiting for the result of the right hand side and other callbacks will be given a chance to execute in the mean time."

That will let people know without going into a ton of detail that their function execution will get paused and will hopefully be enough to give them some ideas if the notice issues from it about what to look for.

Other than that I think it looks great.

comment:27 Changed 3 years ago by Glyph

Resolution: fixed
Status: newclosed

(In [43016]) Merge inlineCallbacks-howto-1009-2: Document inlineCallbacks

Author: glyph

Reviewer: dstufft

Fixes: #1009

.

comment:28 Changed 3 years ago by Glyph

Thanks for the review, dstufft!

Note: See TracTickets for help on using tickets.