[Twisted-Python] Control the number of parallel system calls

[Chris Miles]

In fact the code I use to perform concurrent processing from a disk- 
based request queue is available here,
http://www.psychofx.com/directory_queue_service/

Feedback welcome.

Cheers,
Chris Miles

On 19 Sep 2007, at 11:57, Chris Miles wrote:

> I've done something very similar.  I control the number of  
> concurrent child processes by simply keeping a counter of how many  
> are active at any one time.  The main loop that polls a queue will  
> only accept new jobs if the counter is less than the configuration- 
> defined limit.
>
> I didn't find anything Twisted-specific to limit the number of  
> forked processes, but it was easy to implement in my own logic.
>
> Cheers,
> Chris
>
> On 19 Sep 2007, at 11:26, Daniel de la Cuesta wrote:
>
>> Hi,
>>
>> I am developing a video conversion server. To process the video  
>> conversion I use the method "getProcessOutput" to call "ffmpeg".
>>
>> The user upload the video using a HTTP POST and each video must be  
>> converted to 8 or 10 output formats.
>>
>> Currently, all the conversions for each video are processed in  
>> parallel, what I want to do is to control the number of parallel  
>> system calls to "ffmpeg". For example I only want 2 processes at  
>> the same time and when one of them finishes the next process start.
>>
>> How can I do that?
>>
>> Is there any function in Twisted that implements that?
>>
>> Thank you
>>