Twisted crashes when parsing XMPP XML stream with Python 3.8

[Jérôme Poisson]

Hello,

When parsing an XML stream with Python 3.8, twisted.words.xish.domish import ExpatElementStream is crashing with the following traceback:

        Traceback (most recent call last):
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/python/log.py", line 103, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/python/log.py", line 86, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
            why = selectable.doRead()
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/internet/tcp.py", line 243, in doRead
            return self._dataReceived(data)
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived
            rval = self.protocol.dataReceived(data)
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/words/xish/xmlstream.py", line 93, in dataReceived
            self.stream.parse(data)
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/words/xish/domish.py", line 797, in parse
            self.parser.Parse(buffer)
          File "/build/python/src/Python-3.8.0/Modules/pyexpat.c", line 407, in StartElement
            
          File "/home/goffi/venvs/sat_principal/lib/python3.8/site-packages/twisted/words/xish/domish.py", line 810, in _onStartElement
            for k, v in attrs.items():
        builtins.RuntimeError: dictionary keys changed during iteration

note that this was working as expected with Python 3.7.

I've made a small script which reproduce the issue, the value in buffer comes from real XMPP stream:

#!/usr/bin/env python3.8

from twisted.words.xish.domish import ExpatElementStream


buf = (b"<?xml version='1.0'?><stream:stream from='sat-pubsub.tazar2.int' xmlns:stream='"
       b"http://etherx.jabber.org/streams' id='c2f2fecf-fb57-4a30-bbfe-0e9b5a587a8e' "
       b"xmlns='jabber:component:accept' xml:lang='en'>")

elem_stream = ExpatElementStream()
elem_stream.parse(buf)
print("test passed")

This makes Twisted unusable for XMPP with Python 3.8, so I set the priority as highest.

Thanks!

[hawkowl]

Hi, thanks for the bug report. We don't officially support Python 3.8 yet, so I've added it to the 'support 3.8' milestone. I'd suggest using 3.7 for the time being :)

[Jérôme Poisson]

I've submitted a fix at ​https://github.com/twisted/twisted/pull/1198

[Jérôme Poisson]

Hello hawkowl, thanks for the feedback, I'll apply the fix in my own code until a Python 3.8 compatible version of Twisted is available, as it seems to be enough for me. I hope the PR I've made is fine.

[Glyph]

Thanks for your contribution. Looks good to me; I'll land it.

[Glyph <glyph@…>]

In a24e934:

Merge pull request #1198 from goffi-contrib/9730-goffi-contrib-fix-xmpp-parsing-python-3.8

Author: goffi-contrib

Reviewer: glyph

Fixes: ticket:9730

Fix parsing of namespaced attributes with Python 3.8 in twisted.words.