[Twisted-Python] New to twisted, need a bit of high-level design help.

Brian L. Troutwine goofyheadedpunk at gmail.com
Wed Jul 4 20:12:54 EDT 2007

Hello all,

I'm new to using twisted and am attempting to implement a client and
server for use a distributed computing environment with a centralized
task server. Clients query the task server for a work list, apply a
bunch of transforms to this work list and then return the transformed
work list back to the server. The server, for its part, has to send
out these work lists, keep track of which client is processing what
(for timeout handling) and collate the transformed work lists so that
it can writing the data out to disk in a sorted manner.

I've spent the better part of a week banging out prototypes, reading
the docs and asking questions on IRC (sorry, dear IRC folks, for all
the timeouts), but the things I've built feel clunky. I have primarily
focused my efforts on using t.s.pb and t.p.amp as I control both ends
of the wire. t.p.amp is probably too limited, though, as the values in
packet hashes must be <= 65535 bytes, too small for my purposes unless
I break my loads into peices, which is not an obvious or easy thing to
do given the structure of my computed data.

t.s.pb's documentation seems to concern itself with examples
involiving client requests, server responses and clients working on
the response with no  communication back to the server and, though
it'd be nicer to work with probably, I can't seem to figure out how to
cleanly implement communication back to the server. My prototype
server is relatively clean, supporting a remote_checkin which clients
call with a valid, unique ID to receive data and a remote_checkout,
which clients call with their computed data to register it with the
server. The clients, however, are a mess, mostly because the client
transforms are not simple and might fail in a multitude of ways. (I
really rather need to keep a reference around to the server so I can
arbitrarily push data toward it).

As my attempts to use twisted for this purpose have been frustrated,
due to my inexperience, I'm sure, would it be possible for someone to
explain how I could use twisted in the construction of a distributed
system with a central data collection and collation server? Any
comments are greatly appreciated, from critiques on the  bits of
twisted I've attempted to use to suggestions for new areas to explore
to code samples. (I must apologize that I cannot give any of my
prototype code as I do not own the copyright on it.) I've worked myself
into rather a rut and need help climbing out.


More information about the Twisted-Python mailing list