[Twisted-Python] Request for help with Twisted bindings in M2Crypto

Glyph Lefkowitz glyph at twistedmatrix.com
Tue Jul 26 00:05:17 MDT 2016


> On Jul 25, 2016, at 3:10 AM, Daniel Sank <sank.daniel at gmail.com> wrote:
> 
> I realize this is not the main point of this thread, but I'd like to make a comment regarding Twisted being opaque.
> 
> > Twisted is open source, so none of the data types are opaque.
> 
> That's a non sequitur. A bunch of open source text in a language you don't understand is opaque, or perhaps better called "obscure". Among other things, Twisted's use of interfaces makes the code very hard to understand.

This is not the first time that someone has made this comment, and I find it very discouraging.

An interface is a very simple concept - an abstract description of what an object is expected to provide if you're going to do something useful with it.  As a specific programmatic instantiation of this concept, zope.interface provides us with two chief advantages, one purely informational and one useful at run-time.  The informational one is that rather than saying "this parameter must provide four methods, the first of which is makeConnection, the second of which is (optionally) connectionMade, the third of which is dataReceived, the fourth of which is connectionLost, which take parameters, respectively,..." we can say "@type: L{IProtocol} provider".  The programmatic advantage is that we can ask the question directly; not 'do you have attributes with these names' or 'does this function signature match', but IProtocol.providedBy(something), which asks if 'something' even _intends_ to implement all the relevant functionality described by IProtocol.

In my view, interface definition is the primary activity of software development; deciding how the pieces of the system fit together and precisely describing what they do.  The fact that so many people seem to find either the basic idea of an abstract type, or the concrete instantiation of that idea in the Zope Interface library, so horribly confusing, makes me despair of ever communicating the actually hard stuff that Twisted gets up to in its internals.

The main thing that I have heard in the past about what makes interfaces confusing is that people want to trace through the implementation to see what twisted is 'really doing', and the fact that there is -more than one- implementation of a method like 'listenTCP' is the source of the real confusion, underlying the problem with "interfaces".  But, that's the whole point: 'listenTCP' is an abstract concept.  The better parts of twisted are _more_ abstract, like 'IStreamServerEndpoint.listen', which has dozens of implementations rather than just 3 or 4, or Deferred.callback, which might do anything at all - if the callback chain did one specific thing there would hardly be any point.

So, this is more just an emotional appeal from me, than anything the project needs, but I would very much like to understand *what* is so confusing about "interfaces".  Is it, as Cory posited, just that the documentation is not properly linked?  Or is it that the average Python developer needs a gentle introduction to the entire idea of abstract rather than concrete types?  If they do - is it really Twisted's responsibility to provide it to them?  Should Zope Interface just have a snazzier website?

> > Twisted is probably the best open source project I have worked with
> > in terms of having documentation which is generated from the code
> 
> I agree that the documentation is generally excellent and that the tests coverage is similarly excellent. However, I still find large fractions of the code very hard to comprehend. A while ago I made a serious effort to understand PB and fix some bugs, but the interface stuff combined with some very odd contortions of python class innards eventually lead me to give up. This is despite the friendly helpful attitude of the main developers both here and in IRC (seriously, thanks everyone for your help back then!).

Given that Twisted is often translating network protocol data into Python method calls, one needs both a working domain knowledge of the protocol involved and a robust understanding of Python metaprogramming constructs.  It sounds here like where you fell down was mostly in the "Python metaprogramming" area, where PB is especially intense.  Unfortunately, framework code just looks like that (the inner guts of Django are similarly, if not more, obscure, for example); it's not especially specific to Twisted.

Don't get me wrong, Python metaprogramming _is_ hard.  It's something that I know pretty well, but I can recognize that each additional layer of indirection is additional complexity for someone to learn, and metaprogramming is by definition at least 3 layers indirected from your actual problem.  But it's also not really specific to Twisted either.  This is another case where I'm not sure what to do except to refer people to the language reference and tell them to work through it slowly.

> I just randomly clicked through the docs to this: https://twistedmatrix.com/documents/current/api/twisted._threads.IWorker.html <https://twistedmatrix.com/documents/current/api/twisted._threads.IWorker.html>
> 
> Note that:
> 
> 1. It is an interface, and I still don't _really_ understand what that means in Twisted.

The interface stuff in Twisted is a wholly separate library, 'zope.interface'.  You can read its documentation - both narrative tutorials and API documentation - here:

https://docs.zope.org/zope.interface/README.html

This is referenced by the Twisted documentation which Cory referred to earlier, but I think it would be helpful to call that out specifically - you can read about interfaces completely separate from Twisted.  They don't mean anything special within Twisted itself (beyond our addition of registerAdapter, which is used less and less often in modern Twisted apps).

> 2. I have no idea what a "task" is. I realize this is python and yay duck-typing but not specifying the expected behavior of an argument seems like a big omission.

Did you miss the part where it said "type: 0-argument callable" in the documentation?  The expected behavior of the argument is that it is a thing that can be called, and it takes 0 arguments.  Its return value is unspecified because IWorker providers aren't allowed to use its return value.

> So, while the Twisted docs are great, consider not faulting people for being confused/daunted.

I don't want to fault people, and there are many issues we can address within Twisted's documentation.  For example, culling things from the API documentation and presenting it as it is really intended to be consumed, eliding private methods that have underscores or live in test_* packages to present a smaller surface, improving tutorial documentation, and so on.  It's not entirely the user's fault.

But certain things - the fact that we might have an abstract interface with multiple concrete implementations, for example - are inherent parts of the problem domain that Twisted is trying to address, and this is what Interfaces let us express.  When people have said things like "Twisted's use of interfaces makes 

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20160725/933fed60/attachment-0002.html>


More information about the Twisted-Python mailing list