[Twisted-Python] i want to update 1000 devices running as servers
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Fri Jul 20 06:12:28 MDT 2012
On 07:01 am, luka.rahne at gmail.com wrote:
>I have like 1000 devices that need update that takes like 1 minute
>each. I
>want them to update as quick as possible, sending lines from sx file,
>using
>UDP and for each line i got reponse "OK".
>
>Here is code that works for up to 500 servers, but not so well when i
>go
>more (it just does not finish)
One problem you need to consider is that all platforms limit the number
of simultaneous file descriptors (sockets) you may use at once. On
Windows, this limit is around 500 by default. You can raise it to a
much larger number (but I forget what) by using Twisted's IOCP reactor
instead of the default select reactor.
See the reactor selection howto on the website for details.
>my current simplified client and server_emulator code is here:
>https://gist.github.com/3144857
Another problem with this code is the while loop inside sendDatagram.
Twisted is a cooperative multitasking system. You cannot execute large
loops like that and have everything work properly or well.
Programs based on Twisted do a little bit of work, give up execution
control, then do a little bit more work when they regain execution
control. The work is event-driven, so (for example) perhaps you want to
replace the while loop with code that sends the next datagram when the
previous datagram is acknowledged.
>
>currently i have only one server for simulation, but in real world it
>will
>be 1000 different IP-s.
>
>I was trying to stop and run reactor but did not work (reactor is not
>rerunnable exception or somthing like that).
Yes. You can only call reactor.run() once. That shouldn't be a very
important limitation. You can do as much networking as you want while
only running the reactor once.
>I want to run lets say 100 updates at time and once some update
>is finished i want to remove this protocol out of job and schedule new
>one.
Take a look at twisted.internet.task.cooperate. Here's an example
demonstrating its use to do a limited amount of work concurrently:
http://as.ynchrono.us/2006/05/limiting-parallelism_22.html
>Can somebody give some pointers?
>
>I am running this on windows if that is an issue.
Jean-Paul
More information about the Twisted-Python
mailing list