<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 24, 2017, at 12:48 PM, Craig Rodrigues <<a href="mailto:rodrigc@crodrigues.org" class="">rodrigc@crodrigues.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone <span dir="ltr" class=""><<a href="mailto:exarkun@twistedmatrix.com" target="_blank" class="">exarkun@twistedmatrix.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-">On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <span dir="ltr" class=""><<a href="mailto:rodrigc@crodrigues.org" target="_blank" class="">rodrigc@crodrigues.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote"><div class=""><div class="gmail-m_-4866087461406899210gmail-h5"><div class=""><br class=""></div></div></div>I did some more debugging. The callstack is quite deep. :/<br class="">I used this command to trigger the error:<br class=""><br class="">trial buildbot.test.unit.test_proce<wbr class="">ss_buildrequestdistributor.Tes<wbr class="">tMaybeStartBuilds.test_slow_db<br class=""><br class="">I found in this function in buildbot's BuildRequestsEndpoint.get() function ( <a href="https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146" target="_blank" class="">https://github.com/buildbot/bu<wbr class="">ildbot/blob/master/master/buil<wbr class="">dbot/data/buildrequests.py#<wbr class="">L146</a> ) there is this line:<br class=""><br class=""><b class=""> defer.returnValue(<br class=""> [(yield self.db2data(br)) for br in buildrequests])<br class=""></b><br class="">On Python 2, this line returns:</div><div class="gmail_quote"> an object list</div><div class="gmail_quote"> each list entry is a dictionary of name/value pairs that looks like:</div><div class="gmail_quote"><p class="gmail-m_-4866087461406899210gmail-m_920830734858214402gmail-p1"><span class="gmail-m_-4866087461406899210gmail-m_920830734858214402gmail-s1">[{'buildrequestid': 10, 'complete': False, 'waited_for': False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6, 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for': False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13, 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}]</span></p></div><div class="gmail_quote"><br class=""></div><div class="gmail_quote">On Python 3, this returns:</div><div class="gmail_quote"> an object <generator object BuildRequestsEndpoint.get.<loc<wbr class="">als>.<listcomp></div><div class="gmail_quote"> of type <class 'generator'><br class=""><br class=""></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">Yep.</div><div class=""><br class=""></div><div class="">On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list comprehension. It will have len(buildrequests) elements and each will be the value sent back in to the generator via the yield expression.</div><div class=""><br class=""></div><div class="">On Python 3, the same expression is a list of one element which is a generator expression.</div><div class=""><br class=""></div><div class="">This form is much clearer, I think, and behaves as intended on both versions of Python:</div><div class=""><br class=""></div><div class=""> results = []</div><div class=""> for br in buildrequests:</div><div class=""> results.append((yield self.db2data(br)))</div><div class=""> defer.returnValue(results)</div><span class="gmail-HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class=""><br class=""></div></font></span></div></div></div></blockquote><div class=""><br class=""></div><div class="">Wow, thanks! That fixed it. I submitted those fixes upstream to buildbot.</div><div class=""><br class=""></div><div class=""> I'm not so familiar with generator expressions and Deferred's so have been playing</div><div class="">around with things to try to understand.</div><div class=""><br class=""></div><div class="">I went back to the problem code, and changed:</div><div class=""><br class=""></div><div class=""><b class=""> defer.returnValue(<br class=""> [(yield self.db2data(br)) for br in buildrequests])</b></div><div class=""><br class=""></div><div class="">to:</div><div class=""><br class=""></div><div class=""><b class=""> defer.returnValue(</b></div><div class=""><b class=""> list([</b><b class="">(yield self.db2data(br)) for br in buildrequests])</b></div><div class=""><b class=""><br class=""></b></div><div class="">and ran the code under Python 3 and got a return value of:</div><div class=""><br class=""></div><div class=""> [ Deferred ]</div><div class=""><br class=""></div><div class="">instead of:</div><div class=""> [ {......} ]</div><div class=""><br class=""></div><div class="">where the Deferred.results value was the dictionary.</div><div class=""><br class=""></div><div class="">How does Python 2 directly go to returning the dictionary out of the Deferred,</div><div class="">while Python 3 returns the Deferred inside the list? self.db2data() returns a Deferred.</div></div></div></div></div></blockquote><br class=""></div><div>I've encountered this before and quickly worked around it, but I think this might actually be a bug in python 3, or at least its documentation.</div><div><br class=""></div><div>The language docs officially say that a "list display" (which is what I believe we're looking at here) "yields a new list object". But that is not what I see here:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div>>>> [(yield 1) for x in range(10)]</div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><generator object <listcomp> at 0x10cd210f8></div></blockquote><div><br class=""></div><div>That is a comprehension enclosed in square brackets, but I'm getting a generator object out rather than a list.</div><div><br class=""></div><div>Is there a PEP for this that just hasn't updated the language reference?</div><div><br class=""></div><div>-glyph</div><br class=""></body></html>