[Twisted-Python] Event notification and general RPC in Twisted...

[Blair Bethwaite]

Hi all,

I'm thinking about using Twisted to rewrite some communications in a
Grid computing application.
E.g.:

[JobServer]1 ---------- *[Proxy]1 ----------- *[Agents]

Agents get jobs, report status, results, etc through Proxy to
JobServer. Agents are often distributed across a private network that
has no external interface, hence the proxy, which is run on a machine
between the private network and internet. Also, the proxy may do some
caching/queuing of particular messages - especially where the number
of agents behind it is large.

What we have at present is a TCP socket server however we're starting
to hit scalability issues and on top of that our future plans
necessitate a notification framework. I've been digging around on
twistedmatrix.com and in the book but so far am having trouble finding
any guidance or the level of detail that tells me whether Twisted has
what we need, some bits of which are:

- we'd like to use a persistent stream/connection, at least between
the JobServer and Proxy (traffic frequency will be reasonably high)
- it needs to be interoperable with java (is there PB for java?)
- each end of the connection should be able to register for and get
notifications from the other (e.g. Agent gives a heartbeat, JobServer
tells Agent to stop)
- sometimes the Proxy might be behind a firewall and only able to
connect out, we need to be able to use that connection to go back as
above
- we want to dynamically configure streaming connections between Proxies

Hopefully that's enough context for you to point at modules and tell
whether these are appropriate questions:
Can I re-use a TCP stream for multiple XML-RPC or PB operations?
If so, would it make sense to have the client (e.g. Proxy or Agent)
initiate the connection and then make a rpc (e.g. "notify me of
anything relevant") to the server which would eventually return when
something relevant came along - triggering a callback in the client...
wash, rinse, repeat?

Sorry if I'm overlooking something that already answers this, it
doesn't seem like something too unique to have not been asked
before...

Regards,
-Blair

-- 
In science one tries to tell people, in such a way
as to be understood by everyone, something that
no one ever knew before. But in poetry, it's the
exact opposite.
 - Paul Dirac