[Twisted-Python] Two main loops

[Jasper]

First off, thanks for the reply!  My responses/questions are inline below.

-Jasper

glyph at divmod.com wrote:
>
> On 09:06 pm, jasper at peak.org wrote:
>> That depends.  When I send moderately sized objects over Twisted, 
>> this approach causes my frame rate to stop while Twisted is busy, and 
>> worse makes my GUI non-interactive.  For a game, this is generally 
>> unacceptable...
> You can't send "objects" over "Twisted".

Sure, not per se.  However, you can essentially send copies of objects 
through PB, provided both sides of the wire have the same class definition.


> Are you talking about a particular protocol in Twisted?  Serializing 
> large objects through PB might take a long time.  In that case, it's 
> not "Twisted" that is busy, but your program which is busy serializing 
> that object.

It comes in the twisted installation package, in the module 
"twisted.spread.pb" ;-)

Anyway, the delay comes in a call to (a) perspective.callRemote().  I 
understand that this isn't a reactor scheduled action, but nonetheless I 
don't have any direct control over how things are jellied without 
hacking Twisted.


>> I'm not sure of a good/nice way to deal with this.  Slicing 
>> transmissions into a sequence of smaller chunks seems like a fair 
>> pain in the ass, but then again the alternatives are punting to 
>> another thread/process. :-/
>
> Actually, punting to another thread or process is still going to slow 
> down your rendering, but in a much less deterministic way.  The OS 
> scheduler is going to be interrupting the rendering rather than the 
> long-running serialization function.  Breaking down the serialization 
> work into smaller chunks will allow you to better control what's going 
> on.
>
> Sending the work of serializing a large object to another process 
> won't work anyway: how are you going to get that large object to the 
> other process to serialize in the first place, without serializing it? 
> Sending it to another thread probably won't work either, since if the 
> object is large it is likely shared (and therefore will involve 
> locking large portions of the data structure for long periods of time) 
> and Python's GIL will introduce deleterious effects for your framerate 
> anyway.

In hindsight I see a separate process won't work (duh!)  However, I was 
thinking a thread might work, since the data I'm sending (players get a 
filtered view of the game's true state) isn't used by anything else, and 
the data isn't /that/ large. ;-)  Am I missing something low level 
here?  Would this end up locking bits other than those being serialized?

I suppose it's a bit moot, as I'm reluctant to fiddle with threading.


> Fundamentally what you are doing here is optimizing your program.  
> Your serialization is slow and it needs to be faster so that it does 
> not impact your framerate.  This is a difficult problem, one which is 
> made worse by the fact that Python does not have a particularly fast 
> runtime. There's no quick answer: you don't need to restructure your 
> main loop, you need to think about what your program is spending its 
> time doing.
 
Optimizing my program (e.g. by sending a bunch of smaller data chunks) 
feels like the wrong way to go...  Wouldn't it make more sense to hack 
PB's callRemote() to do this in a more general manner?  I'm thinking 
somewhere around jelly._Jellier.jelly()'s recursive calls to itself;  
perhaps using generators, although I'm fuzzy on the implications of 
recursion + generators...

Either way it's definitely optimization, so not exactly a pressing issue 
for me to resolve until the end.


-Jasper