[Twisted-Python] Cause epollreactor to busyspin

Tim Hughes thughes at thegoldfish.org
Thu Apr 21 11:26:48 MDT 2016


On 2016-04-21 11:30, Tim Hughes wrote:
> On 2016-04-21 03:57, Glyph wrote:
>>> On Apr 20, 2016, at 10:37, Tim Hughes <thughes at thegoldfish.org> 
>>> wrote:
>>> 
>>> Hi All,
>>> 
>>> I have a latency sensitive application that gets affected when it is 
>>> moved between cores or the system has to wake from idling.
>> 
>> Can you explain "gets affected" in more detail?  If moving between
>> cores is the issue, could you set its CPU affinity?
> 
> That is exactly what I would do with a physical machine but
> unfortunately where I need to use it is in china and I only have xen
> virtual machines available. I know this isn't the best situation but
> unfortunately that is all I can get in these locations.
>> 
>>> I would like to run the thread in as tight a loop as possible and to 
>>> do this i need to cause epollreactor to busyspin on receiving data.
>> 
>> I am very curious about your application now :)
> 
> We are using it to test the performance of streaming financial market
> data over different CDN providers to different cities around the
> world. Currently trying to jump through the hoops to be able to
> opensource it and stick it on github. It isn't that exciting, it is
> basically a modified LineReceiver with the delimiter set to \x01 and
> the messages are key=value pairs at a peak rate of approx 130000 pairs
> per second
> https://en.wikipedia.org/wiki/Financial_Information_eXchange
> 

Hoops have been jumped and it is on github at 
https://github.com/LMAX-Exchange/txfixclient


>> 
>>> I have worked out how to do it by editing the twisted code directly 
>>> by hard coding the value of `timeout` to be 0 at this location in the 
>>> code 
>>> https://github.com/twisted/twisted/blob/f074ba3d5083aa1503abcf194aece327e7f84805/twisted/internet/epollreactor.py#L370
>> 
>>> Is it possible to actually set this on the reactor in a more sensible 
>>> way so I don't need to patch the twisted codebase. ? Below is 
>>> basically what I am doing.
>> 
>> You can achieve this with a tiny bit of extra overhead by doing
>> something like this:
>> 
>> def cant_sleep_clown_will_eat_me():
>>     reactor.callLater(0, cant_sleep_clown_will_eat_me)
>> cant_sleep_clown_will_eat_me()
> 
> Will give this a go
> 
>> 
>> Does this actually improve your latency?
>> 
>> -glyph
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python





More information about the Twisted-Python mailing list