[Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?

[Wim Lewis]

TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?

I don't have time right now to set up a PyPy-capable environment and try 
to reproduce this, but perhaps someone does?

Long version:

Trunk has been broken since the last merge a week or so ago, but I 
don't think the breakage is due to that merge.  As an experiment I made 
a PR based off the last successful version of trunk, with a whitespace 
change, and it now fails CI as well. So I think the failure must be due 
to some change that isn't in Twisted or controlled-for by tox.

The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. 
Unlike our usual CI problems it doesn't seem to be a random failure: it 
fails all the time in the same place. But the place doesn't make sense 
to me. It's in the IRC CTCP tests, and they fail in the same ways each 
time (an expected response is not received).

The pair of CI runs closest to the change are these:

run 5793: https://github.com/twisted/twisted/runs/2328450554
run 5809: https://github.com/twisted/twisted/runs/2360415474

There are a lot of differences, but sys.version went from 3.7.9 to 
3.7.10 between those runs, so that seems like the most likely culprit.

Last working:
>  sys.version              : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52)
>  [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
>  sys.prefix               : /opt/hostedtoolcache/PyPy/3.7.9/x64
>  sys.exec_prefix          : /opt/hostedtoolcache/PyPy/3.7.9/x64
>  sys.executable           : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python

First broken:
>  sys.version              : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34)
>  [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
>  sys.prefix               : /opt/hostedtoolcache/PyPy/3.7.10/x64
>  sys.exec_prefix          : /opt/hostedtoolcache/PyPy/3.7.10/x64
>  sys.executable           : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python

PyPy's release notes for 7.3.4 don't list anything that jumps out at me:

https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html

My guess would be some latent timing bug in Twisted that was uncovered 
by pypy execution time changes (I don't imagine that the CTCP code gets 
exercised very heavily these days) or perhaps PyPy got a bug.


Wim.