[Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot

[Craig Rodrigues]

On Tue, Jan 24, 2017 at 12:48 PM, Craig Rodrigues <rodrigc at crodrigues.org>
wrote:

>
>
> On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone <
> exarkun at twistedmatrix.com> wrote:
>
>> On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodrigc at crodrigues.org>
>> wrote:
>>
>>>
>>>
>>>
>>> I did some more debugging.  The callstack is quite deep. :/
>>> I used this command to trigger the error:
>>>
>>> trial  buildbot.test.unit.test_process_buildrequestdistributor.Tes
>>> tMaybeStartBuilds.test_slow_db
>>>
>>> I found in this function in buildbot's BuildRequestsEndpoint.get()
>>> function ( https://github.com/buildbot/buildbot/blob/master/master/buil
>>> dbot/data/buildrequests.py#L146 ) there is this line:
>>>
>>>
>>>
>>> *        defer.returnValue(            [(yield self.db2data(br)) for br
>>> in buildrequests])*
>>> On Python 2, this line returns:
>>>   an object list
>>>   each list entry is a dictionary of name/value pairs that looks like:
>>>
>>> [{'buildrequestid': 10, 'complete': False, 'waited_for': False,
>>> 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11,
>>> 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6,
>>> 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
>>> 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for':
>>> False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid':
>>> 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13,
>>> 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
>>> 'priority': 0}]
>>>
>>> On Python 3, this returns:
>>>   an object <generator object BuildRequestsEndpoint.get.<loc
>>> als>.<listcomp>
>>>   of type <class 'generator'>
>>>
>>>
>> Yep.
>>
>> On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list
>> comprehension.  It will have len(buildrequests) elements and each will be
>> the value sent back in to the generator via the yield expression.
>>
>> On Python 3, the same expression is a list of one element which is a
>> generator expression.
>>
>> This form is much clearer, I think, and behaves as intended on both
>> versions of Python:
>>
>>     results = []
>>     for br in buildrequests:
>>         results.append((yield self.db2data(br)))
>>     defer.returnValue(results)
>>
>>
>>
> Wow, thanks!  That fixed it.  I submitted those fixes upstream to buildbot.
>
>  I'm not so familiar with generator expressions and Deferred's so have
> been playing
> around with things to try to understand.
>
> I went back to the problem code, and changed:
>
>
> *        defer.returnValue(            [(yield self.db2data(br)) for br in
> buildrequests])*
>
> to:
>
> *       defer.returnValue(*
> *           list([**(yield self.db2data(br)) for br in buildrequests])*
>
> and ran the code under Python 3 and got a return value of:
>
>            [ Deferred ]
>
> instead of:
>            [ {......} ]
>
> where the Deferred.results value was the dictionary.
>
> How does Python 2 directly go to returning the dictionary out of the
> Deferred,
> while Python 3 returns the Deferred inside the list?  self.db2data()
> returns a Deferred.
>
> Thanks.
>
> --
> Craig
>


On the python-dev mailing list, Joe Jevnik gave a really excellent
explanation of what
is going on here, down to the Python bytecode level:

https://mail.python.org/pipermail/python-dev/2017-January/147244.html


--
Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170126/1af9491c/attachment-0002.html>