[Twisted-Python] Cause epollreactor to busyspin

[Tim Hughes]

On 2016-04-21 03:57, Glyph wrote:
>> On Apr 20, 2016, at 10:37, Tim Hughes <thughes at thegoldfish.org> wrote:
>> 
>> Hi All,
>> 
>> I have a latency sensitive application that gets affected when it is 
>> moved between cores or the system has to wake from idling.
> 
> Can you explain "gets affected" in more detail?  If moving between
> cores is the issue, could you set its CPU affinity?

That is exactly what I would do with a physical machine but 
unfortunately where I need to use it is in china and I only have xen 
virtual machines available. I know this isn't the best situation but 
unfortunately that is all I can get in these locations.
> 
>> I would like to run the thread in as tight a loop as possible and to 
>> do this i need to cause epollreactor to busyspin on receiving data.
> 
> I am very curious about your application now :)

We are using it to test the performance of streaming financial market 
data over different CDN providers to different cities around the world. 
Currently trying to jump through the hoops to be able to opensource it 
and stick it on github. It isn't that exciting, it is basically a 
modified LineReceiver with the delimiter set to \x01 and the messages 
are key=value pairs at a peak rate of approx 130000 pairs per second 
https://en.wikipedia.org/wiki/Financial_Information_eXchange

> 
>> I have worked out how to do it by editing the twisted code directly by 
>> hard coding the value of `timeout` to be 0 at this location in the 
>> code 
>> https://github.com/twisted/twisted/blob/f074ba3d5083aa1503abcf194aece327e7f84805/twisted/internet/epollreactor.py#L370
> 
>> Is it possible to actually set this on the reactor in a more sensible 
>> way so I don't need to patch the twisted codebase. ? Below is 
>> basically what I am doing.
> 
> You can achieve this with a tiny bit of extra overhead by doing
> something like this:
> 
> def cant_sleep_clown_will_eat_me():
>     reactor.callLater(0, cant_sleep_clown_will_eat_me)
> cant_sleep_clown_will_eat_me()

Will give this a go

> 
> Does this actually improve your latency?
> 
> -glyph
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python