[Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue 20
Ed Suominen
general at eepatents.com
Mon Dec 17 10:15:45 EST 2007
Yes, it sounds like your time investigating this was well spent.
Asynchronous programming is a whole new way of thinking. And after
programming with Twisted for several years now, I really can't think any
other way; blocking for results just seems backwards somehow.
In case anyone's interested, my own work with CPU-intensive processing
involves Monte Carlo analysis and Bayesian inference of statistical
models. I'm currently performing probabilistic random walks in 6+
dimensional parameter spaces using a cluster of PCs and processor cores.
All of the nodes are managed via my AsynCluster package,
http://foss.eepatents.com/AsynCluster, which is built on AsynQueue and
of course Twisted itself. The main script instantiates a job client that
directs the AsynCluster server to dispatch hundreds of jobs at a time to
the cluster, the deferred results of which are processed with a single
"defer.gatherResults" call for each of thousands of iterations in a
"defer.deferredGenerator" loop.
Some of the nodes are significantly faster than others. That's fine,
because the dispatch queue just feeds more job calls to the faster
nodes. Right now each run of a million or so model evaluations is taking
about an hour apiece on 5-6 nodes. My new 3+ GHz quad-core PC is nearly
done, so things are about to ramp up some more...
By the way, if Twisted Matrix Laboratories wants to use the above as a
success story, they are welcome to do so. Twisted is a triumph of
computer science and the foundation of my current work.
Best regards,
Ed Suominen
http://foss.eepatents.com
Josef Novak wrote:
> Thank you for all the feedback, it was quite useful. I'm still quite
> new to twisted, threads and quite the novice programmer, as my questions
> indicate; this project has been quite enlightening. As a solution I
> decided that passing off the compute intensive operations in a proper
> callback to another (set of) machine(s), which is dedicated to these
> jobs was a more viable longterm solution than this start-stop madness I
> initially came up with. However, attempting it initially resulted in a
> great opportunity for me to learn a bit about threads and processes, so
> it wasn't necessarily a waste of time.
> Thanks again!
> -joe
>
> 2007/12/17, twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com> <
> twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com>>:
>
> Send Twisted-Python mailing list submissions to
> twisted-python at twistedmatrix.com
> <mailto:twisted-python at twistedmatrix.com>
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> or, via email, send a message with subject or body 'help' to
> twisted-python-request at twistedmatrix.com
> <mailto:twisted-python-request at twistedmatrix.com>
>
> You can reach the person managing the list at
> twisted-python-owner at twistedmatrix.com
> <mailto:twisted-python-owner at twistedmatrix.com>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Twisted-Python digest..."
>
>
> Today's Topics:
>
> 1. Re: Re: Twisted-Python Digest, Vol 45, Issue 18
> (glyph at divmod.com <mailto:glyph at divmod.com>)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 16 Dec 2007 10:13:08 -0000
> From: glyph at divmod.com <mailto:glyph at divmod.com>
> Subject: Re: [Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue
> 18
> To: twisted-python at twistedmatrix.com
> <mailto:twisted-python at twistedmatrix.com>
> Message-ID:
> <
> 20071216101308.21558.555241615.divmod.xquotient.2970 at joule.divmod.com
> <mailto:20071216101308.21558.555241615.divmod.xquotient.2970 at joule.divmod.com>>
>
> Content-Type: text/plain; charset="utf-8"; format="flowed"
>
> On 10:04 am, josef.robert.novak at gmail.com
> <mailto:josef.robert.novak at gmail.com> wrote:
> >Ok I see this will definitely not work as we are talking about threads
> >and
> >not processes so os.kill is of no use to me. I guess there is no way
> >to
> >track these new threads and stop/pause/restart them based on some
> other
> >decision logic? Would have been nice if there were though... -joe
>
> If you really think about it (and I mean really think about it, computer
> science-y stuff, understanding what sharing state in threads really
> means, how the Python interpreter happens to work now, considering all
> the ways it could work) the idea of stopping and restarting threads is
> inherently broken. You might be able to make it work, by accident, for
> a few simple cases, but there would be no reasonable way to guarantee
> safety except for forcing the explicit use of channels to communicate
> between threads. If you're doing that, though, you're effectively
> using
> processes. SIGSTOP and friends work just fine on them :).
>
>
>
> ------------------------------
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> <mailto:Twisted-Python at twistedmatrix.com>
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> End of Twisted-Python Digest, Vol 45, Issue 20
> **********************************************
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
More information about the Twisted-Python
mailing list