[Twisted-Python] Job server

Umut Aydin devnetwork at gmail.com
Tue Dec 21 06:48:12 MST 2010


I will write a README and INSTALL documents asap but I did not have
anything about USAGE document in my mind. I will take care of it also.
Thank you for your advice.

Migrating a job during processing is not available yet but I think I
can find a way to provide that within Multiple Job Handler but I dont
have any idea at the moment. Multiple Job Handler idea is based on
separating jobs into small pieces with a job wizard tool. I know that
sounds very limited but I will make it flexible as much as I can then
people will be available to add their own patterns. For example; lets
say we have an uploaded photo and we need to resize it and save into
our storage then we need to update database with this new information.
In that case, Multiple Job Handler sends this job to different workers
with related information and collects the answers for any necessary
response. If anything goes wrong with these workers Multiple Job
Handler sends broken part(s) of the main request to another server.
That s just came into my mind right now. That s not exactly a
migration process but could be work.

Actually for now, I want to create a job farm around a single jobbar
server and sync them with other farms. In that way all servers would
serve all jobs in whole platform. That s what I m trying to do first.
The missing part here is that pushing forward on wrong remote call
requests. That means if we couldnt find any worker connection when we
got a remote call, we should push it to another server with another
remote call. I ll add that asap. This is kind of a hobby for me and I
m pretty busy with my regular job these days :(

I hope I explain the idea in my mind clearly.


On Tue, Dec 21, 2010 at 2:33 PM, Yaroslav Fedevych
<jaroslaw.fedewicz at gmail.com> wrote:
> On Tue, Dec 21, 2010 at 1:40 PM, Umut Aydin <devnetwork at gmail.com> wrote:
>> Hello everybody
>> My job server is on progress and maybe you want to check it out.
>> Distribution part is ready but I still work on remote calls. As soon
>> as I get it done I ll write an article about building a job server
>> with twisted framework. (I wrote an article about Apache Mina
>> framework while designing jobbar
>> http://umut.mobi/blog/2010/11/11/how-to-build-a-distributed-job-server-with-apache-mina-framework/)
>> I m pretty busy these days it'll take almost a week more. Here is
>> website of the project
>> http://jobbarserver.org/
>> By the way, my goal with this project is building a distributed job
>> server and useful extensions for needs of any kind of projects. Some
>> extensions in my mind:
>> Memory Cache, Document Storage, Multiple Job Handler for parallel
>> processing, Log Manager, Data Traffic  Analyser
>> If you have any idea please share with me.
> Before proceeding with any extensions, would be extremely nice to have:
> a README with a description of what it is and what do I gain from
> using it right now.
> an INSTALL with a description of how to make it available systemwide,
> how to ship it with my project or what might break if I install it.
> a USAGE with a simple boilerplate with its bits explained: what do I
> need this code for? What bad happens if I take this bit out? What
> stuff can I customize and with what degree of freedom?
> *.pyc in your .gitignore
> docstrings for classes
> Someone with more profound knowledge would surely tell you about unit tests.
> And my question is, how do I migrate a job to another server? I always
> thought that this is what "distributedness" is about...
> --
> Engineer : How do I do it?
> Economist : How much will it cost?
> Twisted Developer: But does it block?
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Best Regards,

Email: me at umut.mobi

Skype: umut-aydin

Blog: http://umut.mobi

More information about the Twisted-Python mailing list