[Twisted-Python] i want to update 1000 devices running as servers

Laurens Van Houtven _ at lvh.cc
Fri Jul 20 03:51:05 EDT 2012


What version of twisted are you using and which reactor is it using? (You can tell from the logs)

Note that your process for getting data from the datasource is blocking. What sort of data source is it, and why are you stripping stuff off of it? You might want to consider something more cooperative, such as for example a file producer:

http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.FileSender.html

I'm assuming you are aware of all of the issues involving UDP (specifically, that in the real world your file is probably not going to make it to the other side)

The basic pattern will probably be something like:
1) create 100 deferreds that will fire when you're done transferring
2) gatherResults/DeferredList to wait on all of them
3) do (1) again in a callback to that deferred list until you run out of clients

Twisted might have some standard tool somewhere to implement this, but I don't know about it.

cheers
lvh


On 20 Jul 2012, at 09:01, Luka Rahne wrote:

> I have like 1000 devices that need update that takes like 1 minute each. I want them to update as quick as possible, sending lines from sx file, using UDP and for each line i got reponse "OK".
> 
> Here is code that works for up to 500 servers, but not so well when i go more (it just does not finish)
> 
> my current  simplified client and server_emulator code is here:
> https://gist.github.com/3144857
> 
> 
> currently i have only one server for simulation, but in real world it will be 1000 different IP-s.
> 
> I was trying to stop and run reactor but did not work (reactor is not rerunnable exception or somthing like that).
>  I want to run lets say 100 updates at time and once some update is finished i want to remove this protocol out of job and schedule new one.
> 
> Can somebody give some pointers?
> 
> I am running this on windows if that is an issue.
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




More information about the Twisted-Python mailing list