[Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot

[Jean-Paul Calderone]

On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodrigc at crodrigues.org>
wrote:

>
>
> On Sat, Jan 21, 2017 at 8:06 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>
> wrote:
>
>>
>> On Jan 21, 2017, at 6:15 PM, Craig Rodrigues <rodrigc at crodrigues.org>
>> wrote:
>>
>> If I run the test on Python 2, I don't get the error, and on line 93,
>> brdicts is a dict.
>> However, if I run the test on Python 3, brdicts is a Deferred, thus the
>> error.
>>
>>
>> This really ought to be impossible.  If I were seeing this,
>> single-stepping through inlineCallbacks in pudb would probably be my next
>> step.  Have you made any attempts to simplify it down to remove some of the
>> buildbot-specific code?
>>
>> -glyph
>>
>>
>
> I did some more debugging.  The callstack is quite deep. :/
> I used this command to trigger the error:
>
> trial  buildbot.test.unit.test_process_buildrequestdistributor.
> TestMaybeStartBuilds.test_slow_db
>
> I found in this function in buildbot's BuildRequestsEndpoint.get()
> function ( https://github.com/buildbot/buildbot/blob/master/master/
> buildbot/data/buildrequests.py#L146 ) there is this line:
>
>
>
> *        defer.returnValue(            [(yield self.db2data(br)) for br in
> buildrequests])*
> On Python 2, this line returns:
>   an object list
>   each list entry is a dictionary of name/value pairs that looks like:
>
> [{'buildrequestid': 10, 'complete': False, 'waited_for': False,
> 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11,
> 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6,
> 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
> 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for':
> False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid':
> 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13,
> 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
> 'priority': 0}]
>
> On Python 3, this returns:
>   an object <generator object BuildRequestsEndpoint.get.<
> locals>.<listcomp>
>   of type <class 'generator'>
>
>
Yep.

On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list
comprehension.  It will have len(buildrequests) elements and each will be
the value sent back in to the generator via the yield expression.

On Python 3, the same expression is a list of one element which is a
generator expression.

This form is much clearer, I think, and behaves as intended on both
versions of Python:

    results = []
    for br in buildrequests:
        results.append((yield self.db2data(br)))
    defer.returnValue(results)

Jean-Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170123/23845ae8/attachment-0002.html>