[Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot

Glyph Lefkowitz glyph at twistedmatrix.com
Tue Jan 24 14:14:32 MST 2017


> On Jan 24, 2017, at 12:48 PM, Craig Rodrigues <rodrigc at crodrigues.org> wrote:
> 
> 
> 
> On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone <exarkun at twistedmatrix.com <mailto:exarkun at twistedmatrix.com>> wrote:
> On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodrigc at crodrigues.org <mailto:rodrigc at crodrigues.org>> wrote:
> 
> 
> 
> I did some more debugging.  The callstack is quite deep. :/
> I used this command to trigger the error:
> 
> trial  buildbot.test.unit.test_process_buildrequestdistributor.TestMaybeStartBuilds.test_slow_db
> 
> I found in this function in buildbot's BuildRequestsEndpoint.get() function ( https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146 <https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146> ) there is this line:
> 
>         defer.returnValue(
>             [(yield self.db2data(br)) for br in buildrequests])
> 
> On Python 2, this line returns:
>   an object list
>   each list entry is a dictionary of name/value pairs that looks like:
> [{'buildrequestid': 10, 'complete': False, 'waited_for': False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6, 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for': False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13, 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}]
> 
> 
> On Python 3, this returns:
>   an object <generator object BuildRequestsEndpoint.get.<locals>.<listcomp>
>   of type <class 'generator'>
> 
> 
> Yep.
> 
> On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list comprehension.  It will have len(buildrequests) elements and each will be the value sent back in to the generator via the yield expression.
> 
> On Python 3, the same expression is a list of one element which is a generator expression.
> 
> This form is much clearer, I think, and behaves as intended on both versions of Python:
> 
>     results = []
>     for br in buildrequests:
>         results.append((yield self.db2data(br)))
>     defer.returnValue(results)
> 
> 
> 
> Wow, thanks!  That fixed it.  I submitted those fixes upstream to buildbot.
> 
>  I'm not so familiar with generator expressions and Deferred's so have been playing
> around with things to try to understand.
> 
> I went back to the problem code, and changed:
> 
>         defer.returnValue(
>             [(yield self.db2data(br)) for br in buildrequests])
> 
> to:
> 
>        defer.returnValue(
>            list([(yield self.db2data(br)) for br in buildrequests])
> 
> and ran the code under Python 3 and got a return value of:
> 
>            [ Deferred ]
> 
> instead of:
>            [ {......} ]
> 
> where the Deferred.results value was the dictionary.
> 
> How does Python 2 directly go to returning the dictionary out of the Deferred,
> while Python 3 returns the Deferred inside the list?  self.db2data() returns a Deferred.

I've encountered this before and quickly worked around it, but I think this might actually be a bug in python 3, or at least its documentation.

The language docs officially say that a "list display" (which is what I believe we're looking at here) "yields a new list object".  But that is not what I see here:

>>> [(yield 1) for x in range(10)]
<generator object <listcomp> at 0x10cd210f8>

That is a comprehension enclosed in square brackets, but I'm getting a generator object out rather than a list.

Is there a PEP for this that just hasn't updated the language reference?

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20170124/4302750e/attachment.html>


More information about the Twisted-Python mailing list