[Twisted-Python] i want to update 1000 devices running as servers

[exarkun at twistedmatrix.com]

On 07:01 am, luka.rahne at gmail.com wrote:
>I have like 1000 devices that need update that takes like 1 minute 
>each. I
>want them to update as quick as possible, sending lines from sx file, 
>using
>UDP and for each line i got reponse "OK".
>
>Here is code that works for up to 500 servers, but not so well when i 
>go
>more (it just does not finish)

One problem you need to consider is that all platforms limit the number 
of simultaneous file descriptors (sockets) you may use at once.  On 
Windows, this limit is around 500 by default.  You can raise it to a 
much larger number (but I forget what) by using Twisted's IOCP reactor 
instead of the default select reactor.

See the reactor selection howto on the website for details.
>my current  simplified client and server_emulator code is here:
>https://gist.github.com/3144857

Another problem with this code is the while loop inside sendDatagram. 
Twisted is a cooperative multitasking system.  You cannot execute large 
loops like that and have everything work properly or well.

Programs based on Twisted do a little bit of work, give up execution 
control, then do a little bit more work when they regain execution 
control.  The work is event-driven, so (for example) perhaps you want to 
replace the while loop with code that sends the next datagram when the 
previous datagram is acknowledged.
>
>currently i have only one server for simulation, but in real world it 
>will
>be 1000 different IP-s.
>
>I was trying to stop and run reactor but did not work (reactor is not
>rerunnable exception or somthing like that).

Yes.  You can only call reactor.run() once.  That shouldn't be a very 
important limitation.  You can do as much networking as you want while 
only running the reactor once.
>I want to run lets say 100 updates at time and once some update
>is finished i want to remove this protocol out of job and schedule new 
>one.

Take a look at twisted.internet.task.cooperate.  Here's an example 
demonstrating its use to do a limited amount of work concurrently:

http://as.ynchrono.us/2006/05/limiting-parallelism_22.html
>Can somebody give some pointers?
>
>I am running this on windows if that is an issue.

Jean-Paul