[Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot

[Craig Rodrigues]

On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone <
exarkun at twistedmatrix.com> wrote:

> On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodrigc at crodrigues.org>
> wrote:
>
>>
>>
>>
>> I did some more debugging.  The callstack is quite deep. :/
>> I used this command to trigger the error:
>>
>> trial  buildbot.test.unit.test_process_buildrequestdistributor.Tes
>> tMaybeStartBuilds.test_slow_db
>>
>> I found in this function in buildbot's BuildRequestsEndpoint.get()
>> function ( https://github.com/buildbot/buildbot/blob/master/master/buil
>> dbot/data/buildrequests.py#L146 ) there is this line:
>>
>>
>>
>> *        defer.returnValue(            [(yield self.db2data(br)) for br
>> in buildrequests])*
>> On Python 2, this line returns:
>>   an object list
>>   each list entry is a dictionary of name/value pairs that looks like:
>>
>> [{'buildrequestid': 10, 'complete': False, 'waited_for': False,
>> 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11,
>> 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6,
>> 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
>> 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for':
>> False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid':
>> 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13,
>> 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
>> 'priority': 0}]
>>
>> On Python 3, this returns:
>>   an object <generator object BuildRequestsEndpoint.get.<loc
>> als>.<listcomp>
>>   of type <class 'generator'>
>>
>>
> Yep.
>
> On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list
> comprehension.  It will have len(buildrequests) elements and each will be
> the value sent back in to the generator via the yield expression.
>
> On Python 3, the same expression is a list of one element which is a
> generator expression.
>
> This form is much clearer, I think, and behaves as intended on both
> versions of Python:
>
>     results = []
>     for br in buildrequests:
>         results.append((yield self.db2data(br)))
>     defer.returnValue(results)
>
>
>
Wow, thanks!  That fixed it.  I submitted those fixes upstream to buildbot.

 I'm not so familiar with generator expressions and Deferred's so have been
playing
around with things to try to understand.

I went back to the problem code, and changed:


*        defer.returnValue(            [(yield self.db2data(br)) for br in
buildrequests])*

to:

*       defer.returnValue(*
*           list([**(yield self.db2data(br)) for br in buildrequests])*

and ran the code under Python 3 and got a return value of:

           [ Deferred ]

instead of:
           [ {......} ]

where the Deferred.results value was the dictionary.

How does Python 2 directly go to returning the dictionary out of the
Deferred,
while Python 3 returns the Deferred inside the list?  self.db2data()
returns a Deferred.

Thanks.

--
Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170124/ccb10c1a/attachment-0002.html>