[Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue 20

[Ed Suominen]

Yes, it sounds like your time investigating this was well spent.
Asynchronous programming is a whole new way of thinking. And after
programming with Twisted for several years now, I really can't think any
other way; blocking for results just seems backwards somehow.

In case anyone's interested, my own work with CPU-intensive processing
involves Monte Carlo analysis and Bayesian inference of statistical
models. I'm currently performing probabilistic random walks in 6+
dimensional parameter spaces using a cluster of PCs and processor cores.
All of the nodes are managed via my AsynCluster package,
http://foss.eepatents.com/AsynCluster, which is built on AsynQueue and
of course Twisted itself. The main script instantiates a job client that
directs the AsynCluster server to dispatch hundreds of jobs at a time to
the cluster, the deferred results of which are processed with a single
"defer.gatherResults" call for each of thousands of iterations in a
"defer.deferredGenerator" loop.

Some of the nodes are significantly faster than others. That's fine,
because the dispatch queue just feeds more job calls to the faster
nodes. Right now each run of a million or so model evaluations is taking
about an hour apiece on 5-6 nodes. My new 3+ GHz quad-core PC is nearly
done, so things are about to ramp up some more...

By the way, if Twisted Matrix Laboratories wants to use the above as a
success story, they are welcome to do so. Twisted is a triumph of
computer science and the foundation of my current work.

Best regards,
Ed Suominen
http://foss.eepatents.com


Josef Novak wrote:
> Thank you for all the feedback, it was quite useful.  I'm still quite
> new to twisted, threads and quite the novice programmer, as my questions
> indicate; this project has been quite enlightening. As a solution I
> decided that passing off the compute intensive operations in a proper
> callback to another (set of) machine(s), which is dedicated to these
> jobs was a more viable longterm solution than this start-stop madness I
> initially came up with.  However, attempting it initially resulted in a
> great opportunity for me to learn a bit about threads and processes, so
> it wasn't necessarily a waste of time.
> Thanks again!
> -joe
> 
> 2007/12/17, twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com> <
> twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com>>:
> 
>     Send Twisted-Python mailing list submissions to
>             twisted-python at twistedmatrix.com
>     <mailto:twisted-python at twistedmatrix.com>
> 
>     To subscribe or unsubscribe via the World Wide Web, visit
>             
>     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>     or, via email, send a message with subject or body 'help' to
>             twisted-python-request at twistedmatrix.com
>     <mailto:twisted-python-request at twistedmatrix.com>
> 
>     You can reach the person managing the list at
>             twisted-python-owner at twistedmatrix.com
>     <mailto:twisted-python-owner at twistedmatrix.com>
> 
>     When replying, please edit your Subject line so it is more specific
>     than "Re: Contents of Twisted-Python digest..."
> 
> 
>     Today's Topics:
> 
>        1. Re: Re: Twisted-Python Digest, Vol 45, Issue 18
>     (glyph at divmod.com <mailto:glyph at divmod.com>)
> 
> 
>     ----------------------------------------------------------------------
> 
>     Message: 1
>     Date: Sun, 16 Dec 2007 10:13:08 -0000
>     From: glyph at divmod.com <mailto:glyph at divmod.com>
>     Subject: Re: [Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue
>             18
>     To: twisted-python at twistedmatrix.com
>     <mailto:twisted-python at twistedmatrix.com>
>     Message-ID:
>             <
>     20071216101308.21558.555241615.divmod.xquotient.2970 at joule.divmod.com
>     <mailto:20071216101308.21558.555241615.divmod.xquotient.2970 at joule.divmod.com>>
> 
>     Content-Type: text/plain; charset="utf-8"; format="flowed"
> 
>     On 10:04 am, josef.robert.novak at gmail.com
>     <mailto:josef.robert.novak at gmail.com> wrote:
>     >Ok I see this will definitely not work as we are talking about threads
>     >and
>     >not processes so os.kill is of no use to me.  I guess there is no way
>     >to
>     >track these new threads and stop/pause/restart them based on some
>     other
>     >decision logic?  Would have been nice if there were though...  -joe
> 
>     If you really think about it (and I mean really think about it, computer
>     science-y stuff, understanding what sharing state in threads really
>     means, how the Python interpreter happens to work now, considering all
>     the ways it could work) the idea of stopping and restarting threads is
>     inherently broken.  You might be able to make it work, by accident, for
>     a few simple cases, but there would be no reasonable way to guarantee
>     safety except for forcing the explicit use of channels to communicate
>     between threads.  If you're doing that, though, you're effectively
>     using
>     processes. SIGSTOP and friends work just fine on them :).
> 
> 
> 
>     ------------------------------
> 
>     _______________________________________________
>     Twisted-Python mailing list
>     Twisted-Python at twistedmatrix.com
>     <mailto:Twisted-Python at twistedmatrix.com>
>     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> 
>     End of Twisted-Python Digest, Vol 45, Issue 20
>     **********************************************
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
>