[Twisted-Python] Freevo 2.0, Kamaelia, pyevent, Eventnet/LGT: what's going on?

[Valentino Volonghi aka Dialtone]

On Fri, Oct 21, 2005 at 01:10:03AM +0200, Antoine Pitrou wrote:
> 
> Hi,
> 
> Thanks for the answers...

Np :)

> Oh, I have too. But only using the basic networking stuff, because I'm
> not confident enough to touch the high-level things.

I had a bot using twisted.words, Nevow and other projects, that lived long and
prosper for well over an year. Now I changed it because I changed the logging
backend to another database.

> Parts of the Twisted API are marked as "API unstable". I don't think I
> dreamed it, since I've read it in the doc ;)

Sure, I think even the reactor was marked unstable since not so far ago.
Documentation might be not in sync. A problem indeed but still not enough to
drop something and start reimplementing it from scratch. The only way to have
a stable API is when something stops maintainance or completely dies.
Otherwise API changes always.

> Also I've seen several times some answers on this very list which were
> of the sort "actually this module is not maintained anymore" (or
> "experimental").

So? There are tons of experimental projects in python. If something is
experimental, anyway, it doesn't mean that everything in twisted is.
twisted.web is not experimental, twisted.names isn't as well. twisted.mail
too. twisted.spread and so on. those are all very far from being experimental.
And they are not even the only ones.
If something is not maintained anymore it means that the API won't change.
Aren't you happy? :). Anyway bugs can be fixed.

> There is a whole lot of methods which are not documented at all. Also,
> there is usually no comprehensive documentation for a specific
> functionality. Not to mention the annoying interface system which means
> the documentation for e.g. twisted.internet.reactor.ListenTCP is
> actually to be found elsewhere in the class hierarchy...

I can't see how looking inside twisted.internet.interfaces is so hard. Also
python is quite readable and twisted code is cleaner than the code of many
other projects. I've never found any problems with learning how something
works. And indeed I don't know how everything in twisted works. I often open
examples, then I open the source code and so on. We live in the open source
world. Use the Source Luke.

> As I said the problem is that it's not documented anywhere. That doesn't
> mean there isn't any reason, just that a normal developer cannot guess
> by himself. When I see a library with lots of design decisions for which
> I can't find any rationale, I'm legitimately suspicious (it could just
> be the NIH syndrome which we all know very well - since we have all one
> day fallen for it ;-)).

As Itamar said, twisted had logging long before python. Also design decisions
are easily understood by anybody that tried to do the same thing alone. Most
of the projects that deal with event loops are a simplified and poorer version
of twisted. Which means that in the end, if those projects are going to grow
in some way, it's very likely that they will use a similar approach. There is
no magic in Twisted, it's pure python that does what you tell it to, and very
well if I dare say.

> > PB is not proprietary... The protocol is documented (maybe it's the best
> > documented thing in twisted) and source code is available.
> 
> It is proprietary in the sense that it is, AFAIK, implemented only by
> Twisted.

There is a java implementation of pb in the cvs tree. And by this POV I could
say that also OpenDocument is proprietary because it is only implemented by
OpenOffice 2.0.

> Contrast this with e.g. XMLRPC which has lots of implementations. XMLRPC
> is quite mediocre, but at least I know that mostly everyone can find an
> implementation for his/her favourite programming language.

Nobody will tell you to use pb if you don't have the control on both sides of
the connection. NOBODY. What I've seen answered here many times is that you
should prefer pb, because it's better designed for python and for RMI, instead
of XML-RPC or SOAP. But that if you cannot choose what to use on the other
end, then you should stick to the other end choices, or implement pb in the
other language. newpb will make this a bit easier I guess.

> I won't ever use PB for something for which I want others to be able to
> write clients easily (without forcing them to use Twisted). It's a
> simple matter of simplifying interoperability. For some people/projects
> it matters very much, for others it doesn't matter at all.

And nobody has ever suggested to force twisted's pb upon umanity.

> There is a terminology problem, since I don't understand why something
> named "Protocol" should be destroyed at every connection. Or it should
> be, quite logically, renamed "Connection". Also this doesn't appear in
> the API docs (again).

What's the point in having a buildProtocol that re-creates a new protocol for
each connection then? How could the reactor know when to create new protocols
and when not? Why do you believe there is so much magic in twisted? A Protocol
is called like that because it implements a protocol. The fact that it lasts
as long as a connection (another object that already exists in twisted)
doesn't make it different than a protocol implementation.

> For me, a "protocol" is something like HTTP or SMTP. It is "eternal",
> I'm not expecting it to be created/destroyed at every connection (that
> would be like calling an int a float, or whatever).

It is eternal indeed. It's a class with some code in it. How much more
ethernal could you be? instances are created and destroyed for each
connection. Don't mix the things.

> There is no documentation on the magic that really happens when
> something is triggered (for example when a TCP session is established).
> Which methods are called, which objects are created, in which order,
> etc. Unexplained magic is annoying in software development.

In fact there is no magic, why should there be any documentation on something
that there isn't? A framework is cool because mostly you don't need to worry
about what happens below of the abstraction layer you are given.
Why do you think protocols have a connectionMade and a connectionLost method?
When are they called IYHO? Or lineReceived? Or dataReceived? Names are pretty
obvious, trying to put magic where it really isn't is what makes everything
harder. Like with deferreds. They are so simple, and require so little python
knowledge that they have been implemented in javascript too.

What's so hard in a list of functions that is called in a for loop once
their callback method is called?

When you start a TCP connection a socket is added to the reactor. That's all.
That socket has a related connection object that has a related protocol object
(the one you are supposed to write) and a related factory object (by default
unless you change it). data is read from the socket in chunks using the
FileDescriptor abstraction, then it is prepared for the handling of the
protocol that if needed will call the factory.

Pretty straight forward, and very likely what I would do If in the mood to
decide anything.

> It's not that I don't want to learn. It's that the more you use
> framework-specific constructs, and the higher level those constructs
> are, the more your software project is stuck with this framework and
> difficult to reuse/adapt/port/interoperate with. Of course, some
> projects (especially proprietary or internal ones) are less sensitive to
> this than others.

I can see no reason at all. Plenty of stuff in twisted is reusable from
somewhere else. But when coming to networking it just doesn't make any sense.
There is no better alternative, if any then I'm more than happy to look at it
and maybe start helping there instead of here. As all the frameworks when you
use it you are stuck with it. Even with libraries you are stuck with them.
Twisted changes something only because it is async, but that just have
nothing to do with the framework itself. Python sucks because it doesn't allow
async and sync programming style to be merged togheter, this would allow
higher reusability, with the risk of doing something without understanding it
at all. Like not understanding what happens between a:

page = client.getPage('url')

call and the subsequent:

parse(page)

What's the better? Explicit or implicit?

Do we want to talk about simplicity? Python subprocess module, compared to
twisted's reactor.spawnProcess simply sucks. Twisted allows you to open as
many FD's as you want to the child process. subprocess (just created for
python 2.4) only allows 3. I'm involved in a project that right now needs
signature verification and signature creation. We drive gnupg from python. The
twisted backend took less that 40 lines and 30 minutes, and it's perfectly
working.

My boss asked explicitly for a syncronous version of it. The only good
alternative (beyond writing it myself) is using subprocess. Now this took me
over 3 days (and I still need to pass one test) and more than 200 lines of
code. Deep magic and tricks (since I only have 3 FDs to the child process).

My boss has all the rights to ask for a sync version of course. But it's
just too much work, just to avoid using twisted for your very same reason.
Insted of speeding up the project it is just slowing it down.

If by being tied to twisted I gain 3 free payed days of work, then god bless
twisted and who cares if the project now depends on it. Don't you agree?

> Again, that does not change the fact that I'm quite happy with the parts
> of Twisted I do use. The simple notion of a Deferred (with its two
> callbacks) for example, is brilliant.

Most people find magic even in deferreds and start thinking of weird things
right there and stop using twisted only because of them. Just as you stop
using higher levels of twisted (which are the real jewels) only because you
fail to understand them and are not ready to take some time to read the
documentation (when there is) and the source code (which is there for a
reason otherwise twisted would just ship pyc or pyo and live prosper).

Also if you want documentation, start writing some and submit bugs of missing
documentation. We have very little time and most of the free time is already
busy fixing bugs in twisted (which, as all big projects, has bugs too). There
is a documentation team in twisted that was created with the only purpose of
fixing and completing documentation. During the last sprint work has been done
to make it even better. Just contribute, we do like your contributions to the
project and nobody will reject some more documentation. But just saying that
twisted is bad because it has few documentation and stop there, seems like
whining :).

Hope I've been not too harsh at as helpful and informative as possible.

-- 
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.4
Blog: http://vvolonghi.blogspot.com
http://weever.berlios.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
URL: </pipermail/twisted-python/attachments/20051021/9a0ce03c/attachment.sig>