[Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue 20

Doug Farrell dfarrell at mypublisher.com
Mon Dec 17 10:39:16 EST 2007


I've been coding for awhile, but am fairly new to actually using
Twisted. I'm very interested the work you are doing because of a project
I'm thinking of at work. It's not nearly as complicated as your project,
and though we have a number of Linux servers at our disposal, they
aren't clustered at all. We manufacture photobooks, and I'd like to make
our manufacturing workflow more "real time" than it is. What I was
thinking of was building a Twisted backbone for the system and
allocating long term processing elements of the workflow to other
"nodes" using Pyro or PP (the python parallel processing module) tied to
DeferToThread objects. But your solution looks interesting and I'm going
to read more of the link you provide. Any suggestions you might have,
places you could point me for guidance would be greatly appreaciated.

Thanks for the link you provided,
Doug Farrell

-----Original Message-----
From: twisted-python-bounces at twistedmatrix.com
[mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of Ed
Sent: Monday, December 17, 2007 10:16 AM
To: Twisted general discussion
Subject: Re: [Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue

Yes, it sounds like your time investigating this was well spent.
Asynchronous programming is a whole new way of thinking. And after
programming with Twisted for several years now, I really can't think any
other way; blocking for results just seems backwards somehow.

In case anyone's interested, my own work with CPU-intensive processing
involves Monte Carlo analysis and Bayesian inference of statistical
models. I'm currently performing probabilistic random walks in 6+
dimensional parameter spaces using a cluster of PCs and processor cores.
All of the nodes are managed via my AsynCluster package,
http://foss.eepatents.com/AsynCluster, which is built on AsynQueue and
of course Twisted itself. The main script instantiates a job client that
directs the AsynCluster server to dispatch hundreds of jobs at a time to
the cluster, the deferred results of which are processed with a single
"defer.gatherResults" call for each of thousands of iterations in a
"defer.deferredGenerator" loop.

Some of the nodes are significantly faster than others. That's fine,
because the dispatch queue just feeds more job calls to the faster
nodes. Right now each run of a million or so model evaluations is taking
about an hour apiece on 5-6 nodes. My new 3+ GHz quad-core PC is nearly
done, so things are about to ramp up some more...

By the way, if Twisted Matrix Laboratories wants to use the above as a
success story, they are welcome to do so. Twisted is a triumph of
computer science and the foundation of my current work.

Best regards,
Ed Suominen

Josef Novak wrote:
> Thank you for all the feedback, it was quite useful.  I'm still quite
> new to twisted, threads and quite the novice programmer, as my
> indicate; this project has been quite enlightening. As a solution I
> decided that passing off the compute intensive operations in a proper
> callback to another (set of) machine(s), which is dedicated to these
> jobs was a more viable longterm solution than this start-stop madness
> initially came up with.  However, attempting it initially resulted in
> great opportunity for me to learn a bit about threads and processes,
> it wasn't necessarily a waste of time.
> Thanks again!
> -joe
> 2007/12/17, twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com> <
> twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com>>:
>     Send Twisted-Python mailing list submissions to
>             twisted-python at twistedmatrix.com
>     <mailto:twisted-python at twistedmatrix.com>
>     To subscribe or unsubscribe via the World Wide Web, visit
>     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>     or, via email, send a message with subject or body 'help' to
>             twisted-python-request at twistedmatrix.com
>     <mailto:twisted-python-request at twistedmatrix.com>
>     You can reach the person managing the list at
>             twisted-python-owner at twistedmatrix.com
>     <mailto:twisted-python-owner at twistedmatrix.com>
>     When replying, please edit your Subject line so it is more
>     than "Re: Contents of Twisted-Python digest..."
>     Today's Topics:
>        1. Re: Re: Twisted-Python Digest, Vol 45, Issue 18
>     (glyph at divmod.com <mailto:glyph at divmod.com>)
>     Message: 1
>     Date: Sun, 16 Dec 2007 10:13:08 -0000
>     From: glyph at divmod.com <mailto:glyph at divmod.com>
>     Subject: Re: [Twisted-Python] Re: Twisted-Python Digest, Vol 45,
>             18
>     To: twisted-python at twistedmatrix.com
>     <mailto:twisted-python at twistedmatrix.com>
>     Message-ID:
>             <
20071216101308.21558.555241615.divmod.xquotient.2970 at joule.divmod.com
<mailto:20071216101308.21558.555241615.divmod.xquotient.2970 at joule.divmo
>     Content-Type: text/plain; charset="utf-8"; format="flowed"
>     On 10:04 am, josef.robert.novak at gmail.com
>     <mailto:josef.robert.novak at gmail.com> wrote:
>     >Ok I see this will definitely not work as we are talking about
>     >and
>     >not processes so os.kill is of no use to me.  I guess there is no
>     >to
>     >track these new threads and stop/pause/restart them based on some
>     other
>     >decision logic?  Would have been nice if there were though...
>     If you really think about it (and I mean really think about it,
>     science-y stuff, understanding what sharing state in threads
>     means, how the Python interpreter happens to work now, considering
>     the ways it could work) the idea of stopping and restarting
threads is
>     inherently broken.  You might be able to make it work, by
accident, for
>     a few simple cases, but there would be no reasonable way to
>     safety except for forcing the explicit use of channels to
>     between threads.  If you're doing that, though, you're effectively
>     using
>     processes. SIGSTOP and friends work just fine on them :).
>     ------------------------------
>     _______________________________________________
>     Twisted-Python mailing list
>     Twisted-Python at twistedmatrix.com
>     <mailto:Twisted-Python at twistedmatrix.com>
>     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>     End of Twisted-Python Digest, Vol 45, Issue 20
>     **********************************************
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Twisted-Python mailing list
Twisted-Python at twistedmatrix.com

More information about the Twisted-Python mailing list