C based domish.Element serializer

[jack]

For most XMPP applications, XML serialization is the bottleneck. While #2477 improves the Python serializer quite a bit, this can be made faster by dropping down to C.

I've written a Python c module called cserialize which serializes domish.Element objects. This module is compatible with domish and passes most tests. It gives slightly more than a factor of 2 performance increase over the code in #2477.

The failing tests have to do with namespace issues, and after investigating the code, I believe the tests are wrong and need to be rewritten. I haven't rewritten them as I wanted to discuss it first.

You can find the cserialize code here: http://github.com/metajack/cserialize

[jack]

patch to domish.py to use cserialize

[jack]

I just noticed the patch includes an unintended change to the namespace prefixes. Sorry about that.

[ralphm]

First of all, thanks for working on this!

Why not include the C-based serializer in Twisted proper? That would remove the additional external dependency and allow for easier maintenance.

The patch itself is way too complicated. In previous times, there were multiple serializer classes, and they could be switched by setting the public module attribute SerializerClass to point to the serializer class of choice. At some point the even slower _Serializer was removed and _ListSerializer was made the default.

Assuming the inclusion of the c-based serializer as a module, we can just try to import it and have a class _CSerializer (or something) that provides serialize and __init__ like _ListSerializer. If available, it can be assigned to SerializerClass in preference to the pure Python one. An interface definition might be in order.

Could you expand on the failing tests?

[exarkun]

A procedural interjection:

#2477 is already a ticket for XML serialization optimization. This ticket is a duplicate. It's not really important that this ticket was filed for a different set of optimizations. The specific changes which result in fast XML serialization aren't very interesting. What's interesting is that XML serialization is fast. In the light of the 2x improvement of the approach of using a C library instead of a Python library, one thought I have is that the Python optimizations are irrelevant and should be discarded. They obscure the code and don't bring its performance even close to the performance this optimization gives. That said, I'd also really like to explore using an existing C library to do this serialization rather than including a completely custom C extension. However, regardless of what we ultimately decide to do, #2477 is the ticket to associate discussion with.

So, Jack, could you close this ticket as a duplicate and raise the ideas you've presented here on #2477?

[ralphm]

While this ticket is still open, I was reminded about some discussion over at #2741 on the selection of different serializer implementations in domish that should be considered in light of the provided patch.

[exarkun]

I was hoping Jack would notice my comment and agree with my assessment that this is a duplicate or explain why it's not. It's been a couple weeks, though, so I'm just going to call this a duplicate unilaterally.

[jack]

I'm fine with moving the discussion to the other bug. I didn't respond sooner because right after I filed this I got a series of illnesses. I haven't forgotten, and I appreciate your prodding.