[Twisted-Python] Failure of twisted.test.test_plugin.DeveloperSetupTests.test_freshPyReplacesStalePyc under pypy2

[Daniel Sutcliffe]

> On Thu, Jul 13, 2017 at 5:57 AM, Daniel Sutcliffe <dansut at gmail.com> wrote:
>> I shall do my best as my intention is to use Twisted in a live
>> environment with PyPy so it would be great if the project viewed these
>> builds as required to be clean sometime in the future. This was just
>> the first ERROR I attacked, it was always my intention to look into
>> the others too, which I am also seeing.

On Thu, Jul 13, 2017 at 11:31 AM, Craig Rodrigues
<rodrigc at crodrigues.org> wrote:
> Thanks for your motivation.

No problem, this is one of the few ways I can give back to the Twisted
project that is ultimately helping me achieve my goals;
besides getting to the bottom of these type of things is a fun
learning exercise for me :)

> Please keep going, and share your findings on the mailing list.
>
> Last year, Mark Williams worked on fixing many Twisted unit tests to either
> pass or skip on Pypy, and even found one issue with Pypy itself (
> https://bitbucket.org/pypy/pypy/issues/2335/maximum-recursion-depth-exceeded-with
> ).

This one is likely to fall on deaf ears in the PyPy project...
The reason for the ERROR and then FAIL is as I originally thought,
that __import__() under pypy is not loading .pyc files where the .py
is missing and:
  http://doc.pypy.org/en/latest/config/objspace.lonepycfiles.html
tl;dr - they view it as dangerous due to pyc binary variations and
only allow by build time configuration option.

So the next question I guess is how to deal with this:

- presumably my first step is to log a bug in Trac as any change will
require one, I had only held off from doing this until I was more sure
of the cause of the issue.

- the FAIL goes away if the .py is not removed from the plugins dir
and it seems to me that the .py removal is not actually required to
strictly test what
twisted.test.test_plugin.DeveloperSetupTests.test_freshPyReplacesStalePyc
purports to be testing - so that could fix this test

- the fact that the test ERRORs and then carries on and FAILs later in
a strange fashion is due to the fact twisted.plugin.getCache() simply
does a log.err() on encountering the odd situation - so maybe this
could be dealt with better knowing that PyPy does different things
than CPython here... ?

Could the more experienced please advise :)

> If you feel there are improvements or additions to the buildbot jobs using
> Pypy, please share your thoughts.  The buildbot master.cfg file is at
> https://github.com/twisted-infra/braid/tree/master/services/buildbot/master

I have not looked into why buildbot comes up with a different error to
the one I did on various platforms - fix for my situation and see if
it fixes the pypy buildbot before investigating?

The other error that I and the buildbot see on pypy is 12x (always) of
  Failure: twisted.internet.defer.TimeoutError
for
  twisted.protocols.test.test_tls.TLSMemoryBIOTests.test_hugeWrite_TLSv1_1
which is odd as if I run this test on its own it succeeds every time...
Thoughts on this appreciated but probably should start a new thread for that...

Cheers
/dan
-- 
Daniel Sutcliffe <dansut at gmail.com>