[Twisted-Python] Re: Freevo 2.0, Kamaelia, pyevent, Eventnet/LGT: what's going on?

[Michael]

Hi,


I've just come across this thread (unsubbed from the twisted list during 
EuroOSCON, just resubbed after a) getting back b) fixing my machine from a 
reiser corruption...).

Much as Johann Borck in the thread said "I feel like i have to make a 
statement ...", I feel it's probably worth putting in a tuppence worth on the 
Kamaelia comments in the thread. (If only to try and avoid negative 
misunderstandings. I suck though when I try to avoid misunderstandings 
though, so here's fingers crossed I don't cause any here ... )

Also, sorry for the length of this if it's too long. I could reply to the
comments in thread, but that might just cause a thread explosion. If this
is off topic, just delete. Next paragraph probably is probably a sufficient
summary :-)

Some clarification. First of all, I don't see Kamaelia as in competition with 
Twisted. Indeed, Kamaelia should be guttable (people are welcome to
do so), and we'd rather collaborate rather than compete. (I've never
understood the desire to compete in some open source projects, except
on a "for fun"/variety basis)

At all talks longer than my ill-advised lightning talk at Europython 2004 I've 
made it patently clear that I think that Twisted is best of breed in python 
for what it does. And not only that, damn fine at it as well.

The architecture in Kamaelia is however borne from a) working with
architectures similar to, but not quite identical to twisted and wondering
if b) *maybe* there's a better way. I'm not saying Kamaelia *is* that better
way (I'm too close to it), just an alternate approach. That said, if you're
trying an alternate approach, starting from scratch is one way of doing
so, and that's the simple reason we're not using Twisted :-)

After my talk at Europython 2004 I realised to my dismay that some people
mistook my viewpoint of *maybe* there being a better way (badly expressed
then as well :-( ) as being "Twisted sucks/is bad", which is far from the
truth.

Taking some specific points from the thread, which refer to the various 
projects:

Nicola Larosa nico at tekNico.net wrote:
> Why isn't Twisted seen, by these projects at least, as a viable tool? Is
> the event-driven Python world undergoing the same fragmentation as the web
> framework one?

1: Don't ask on the twisted list, ask the people who are running those
    projects :-) (I only discovered the thread because I was resubbing and
    wondered if I'd missed something)
2: We do view twisted as a viable tool. We're just trying something different.
    The "R" part of BBC R&D suggests that trying something different might
    occasionally be a good idea :-) (Then again it might not, experience is
    something you get after you needed it (maybe:) )
3: Kamaelia is much more dataflow driven between active objects rather than
    viewing the system as event driven. Sure, data delivery/receipt are
    clearly events, but that's used for optimisation rather than the core
    idea. (It's also based on recognising that some ways of viewing network
    systems look very similar to asynchronous hardware design)

Maybe they're just different mappings on the same problem space. If so, then 
maybe the different mapping might be a useful different way of looking at the 
problem. **Maybe**.

It's an interesting comment from Freevo's changelog:
   "It is a huge monster and hard to understand"

I think there's an element of truth in that to //any// sufficiently useful 
framework or library. It'd be nice if elements of Kamaelia were able to be 
gutted into twisted, *IF* they'd help with avoiding the above problem. (*IF*)
(A core aim in Kamaelia is to allow novices to get up to learn the core 
quickly)

Stefan Behnel wrote:
> Kamaelia ... is not as 'feature-complete'. So it may get ugly once it
> implements more features. That's a general problem

Indeed, this is possible. However, one interesting thing has shown up
recently - we appear to be converging towards something similar to
deferred generators. Specific example - ask the scheduler to run a
different generator in the place of the current generator, and return
control when done:
    yield WaitComplete(
          self.requestDisplay(DISPLAYREQUEST=True,
                              callback = (self,"control"),
                            size = (width, height),
                            position = self.position
                            )
    )

self.requestDisplay returns a generator, WaitComplete creates an
Axon.Ipc.WaitComplete message, which is passed to the scheduler which
runs it instead of the main component. This then isn't really a huge step to
deferreds/defgens. (If I understand those right) I'm not totally happy with 
the syntax above, and it's likely to change, but it's an initial step.

Itamar Shtull-Trauring itamar at itamarst.org 
> kamaelia: an experimental platform for what I hear are some interesting
> ideas we should look at; 

Please do. Tell us what sucks, take the good stuff. We'll then try and make 
the sucky stuff not suck and then take that :-)

Itamar Shtull-Trauring itamar at itamarst.org 
> probably would be a good candidate for running 
> on top of Twisted since they seem to deal with higher levels of
> abstraction mostly.

Agreed. Not immediately though, we're aiming for around 9-12 months from
now. We should either be running on top of twisted at that point, or side
by side with, or the other way round, or Kamaelia subsumed by something
else.

Antoine Pitrou solipsis at pitrou.net 
> (OTOH, I very briefly looked at Kamaelia's introduction doc and I don't
> find it attractive - this kind of dataflow-programming model looks quite
> unflexible)

I don't suppose you could follow up on that on the Kamaelia list could you?

Moe Aboulkheir moe at divmod.com 
>[ on PEP 342 potentially removing the reliance on frameworks for
>  accomplishing clean asychronous IO - libraries whose clients accrue
>  a dependency on a non-standard reactor/event loop/runtime/whatever
>  in order to operate. kamaelia probably doesn't fit into this category, i
>  wouldn't know. ] 

I think PEP 342 has the potential to simplify everyone's systems. At one point
Kamaelia components could just be run like any other generator. Since then
the dependency on the scheduler has grown somewhat. Prior to seeing this
post I'd already decided that we needed to regrow the earlier simplicity to
allow easier cherry picking by other projects.

Also, personally I'd hope that *if* a similar transform happening in Twisted,
PEP342 may result in is a WSGI for events type situation. That would probably
result in most networked apps being powered by Twisted on some level,
maybe composed in a Kamaelia style way, maybe not. (I've noted the
twisted community have a knack for taking an idea, turning it on it's
head and doing something cool with it :-) )

glyph at divmod.com glyph at divmod.com:

> Also, it gives the Twisted team some potential competition to focus on. 
> Possibly a source of motivation, and besides, everybody loves a race :). 

As long as it's a "Wacky Races" style of competition, I'm happy with Kamaelia
being competition. (Seriously though, Kamaelia is from a "what if", and 
hopefully scavengeable - either now or in future)

> Kamaelia seems more promising to me; I imagine that they might produce some
> serious competition eventually if that's what they aim to do.  

Only in terms of approach. Any sufficiently useful open source project is
easily consumable by another. (Being served up for dinner and eaten is
the ultimate success IMO) I do aim it to be a useful alternative way of 
thinking about problems similar to those that twisted thinks about.

I saw your post here http://www.livejournal.com/users/glyf/41709.html , and
it struck me as interesting. Essentially part of the comments appear to reuse
of generators, which strikes a chord (largely because that's how hardware
would have to work, and Kamaelia's core is based on asynchronous hardware
design[*] ). Again, if we do head down that route (depends on resources, etc),
then I'd hope people feel happy to rip out code where needed.

   [*] Whilst Kamaelia has no chance at present of compiling to silicon,
         things like MyHDL give me hope that it could be possible someday,
         and in that scenario, you need a system similar to the one you
         described.

> However, they claim that only the "Axon" part of their project is stable,
> and that doesn't yet include anything that does select().   

We consider Axon to be API stable (ie stuff that's there is not likely to be 
removed). Axon is designed to be a small framework for handling concurrency, 
communications and composition between (mainly) generators, and threads
have recently been introduced as well. By itself, Axon is intended to be
very, very boring. The instant it's not being boring (such as doing a
select, or poll, or ...) it's doing too much :-)

As a result Axon's version number reflects what we think about it's stability.
The reason for the jump from 1.0.X to 1.1.X was due to threaded components.
(These were added earlier than expected due to a need for threaded
components on Nokia mobiles, and have a suboptimal implementation and
API right now.)

The bulk of Kamaelia however is a collection of components, which is growing
and we can do a large variety of Stuff.

However, we still think (despite having used Kamaelia now for some time), that 
the collection of components we have at present is incomplete (around 30% of 
what would be needed generally speaking), and that the API for the components 
(specifically which inboxes/outboxes they have, names of components, file 
hierarchy etc), is still in flux.  

This collection of Stuff is being driven by application spikes, in application 
areas which are off the beaten track from the project's main goal (to round 
out the system), as well as in the main project goal area. So far this is 
proving to be an interesting approach.

As a result we do have things like TCP server/client components, a selector 
component (which advertises itself as a service internally) which manages a 
select subsystem (which we're thinking of farming off into a thread to allow 
the entire system to sleep waiting for it), pygame Stuff, Tk Stuff, and so 
on. The latest application spike was a simple paint program for example, 
which was useful for testing a variety of subsystems. Similarly for the 
presentation tool spike.

However, the low version number (currently) 0.3.0 is really to act as a 
warning "we still expect quite a lot to change before we hit 1.0.0". I could 
wax lyrical on that, but will only do so if people are interested :) However 
the reason it's 0.3.0, is because at minimum:
   * We want to have sensible integration with twisted (ie Kamaelia usable in
      twisted or vice versa) when we hit 1.0.0
   * There's a whole bunch of core ideas we want to extend before then,
      including the graphical editting toolset.
   * We know we're just skimming the surface of possible components and
      systems (despite the amount of stuff we /can/ do). The component set
      needs to be fleshed out more before we know if the approach has
      worth. (It feels like it to me at present, but then I'm biassed)

That said, generally, things are more stable than unstable, though currently I 
prefer to play up the instability to allow more freedom to experiment with 
APIs and ideas in the short/medium term.

Anyway, I hope that this mail assures everyone on the twisted list (who was 
worried - struck me as a minority) that we're not really in /competition/,
but more looking at providing /options/ for everyone, including Twisted :-)

So far we're finding our approach useful and fun, which at the end of the
day is all that matters. (For work the former is important, if it was just my
personal time, then it could be either :-)

Best Regards,


Michael.
--
Michael.Sparks at rd.bbc.co.uk, http://kamaelia.sourceforge.net/ 
British Broadcasting Corporation, Research and Development 
Kingswood Warren, Surrey KT20 6NP 
 
This message (and any attachments) may contain personal views 
which are not the views of the BBC unless specifically stated.