[Twisted-Python] Using Twisted for distributed computation / experiment running?
robomancer at gmail.com
Tue Apr 3 13:08:05 EDT 2007
> It seems you are wanting a hybrid between a batch queueing system and
> a data management system.
> Any reason not to use something like Torque (nee' OpenPBS) for the
> batch part?
Thanks for the references. Torque seems like a good possibility, but
a bit heavyweight for my tastes; I'd rather have something small and
flexible that I can easily edit to suit my needs. That's why I'd
prefer a solution in Python (whether or not it involves Twisted). I
really am looking only at small-scale stuff -- I have no need for
fault tolerance or scalability beyond maybe 5-10 nodes.
> More info on what your data is like is probably needed for ideas on
> the second part.
All I need is the ability to test several different algorithms on
several different input files. Each algorithm has a variety of
parameter settings, so for every experiment I need to record which
algorithm was used, the parameters, which input file was used, and the
quality of the result (from -1 to 1). I don't think a data management
system is necessary here; I was basically planning on using .csv files
to store experiment settings and results. Again, I'm aiming for
lightweight, not enterprise-grade :)
More information about the Twisted-Python