[Twisted-Python] "3D Game Using Twisted?", or: "Ach du lieber..."

[Glyph Lefkowitz]

On Thu, 2002-03-14 at 17:13, Andrew Bennetts wrote:
> On Thu, Mar 14, 2002 at 01:47:26PM -0800, Derek Simkowiak wrote:
> > -> Well, on the low level, we were discussing make banana extendable so you
> > -> can define custom data types, e.g. a data type for location that is sent 
> > -> in 4 bytes. The overhead for you would be one extra byte per message, I 
> > -> think.
> > 
> > 	Not 4 bytes... 4 BITS (for the Z component only).  I don't know
> > what they're doing, but that's what their Lead Developer told me.
> 
> Yes, but you won't be sending 4 bits as a single packet -- that would have
> just as much latency as 4 bytes, in practice.  The TCP/UDP overhead is more
> like 40 bytes per packet, after all.

And yes, this is where a lot of often-neglected network trickiness
happens -- actually getting more data into each packet.  Balancing
latency against keeping a full packet payload is hard.

> > 	Obviously, the need to fiddle with bitshifting (and other tricks,
> > like compressing certain packet types before transmisison) makes me think
> > I'll need to write my own network library.
> 
> Or extend an existing one...

And, while pb is designed to be extended to do just this, the amount of
application-specific optimization necessary for a first-person shooter
is really phenomenal.  You'll need to optimize PB down quite a bit
further than it's already been (although PB is built to be optimized, so
it's not really a problem) even to be adequate for the control stream. 
The biggest problem I can see you having with the protocol itself is the
fact that it only operates over TCP.  A UDP extension for doing things
like movement packets would be really handy!

> I really know nothing about the internals of twisted.spread, but extending
> it sounds like a better idea than building a new one.

twisted.spread, and Perspective Broker, are both designed to be highly
extensible.  Itamar is currently working on the final iteration of
banana, which will define some mechanisms for backwards-compatibility,
as well as enabling us to write an RFC.

Consider ancillary services that are going to interoperate with your
game, too; if you are extending PB in order to do your game protocol,
then applications like game browsers, high-score charts on webpages (and
so on) will come together in a snap.  Other folks are also writing games
that use Twisted for networking, and it would be neat if they could all
talk to each other!

> > 	There are also further refinements in the CounterStrike/Half-Life
> > version of the Quake2 protocol, although I don't know what those are yet.  
> > I'm also dinking around with ideas like using bits to mark the traversal
> > of a binary "packet opcode" tree, putting the most common types of packets
> > near the top of the tree and the less common (or lower priority) packets
> > near the bottom -- this way, the opcode identifier for the most common
> > packet would only be one bit (and my networking code would be a nasty mess
> > of bitshifting and endian-specific code :).  Given this type of work, I'd
> > rather not just "pass it off" to banana.  (Although I'm still going to
> > take a close look at what's there.)
> 
> Sounds like good ol' Huffman codes to me.  Not a bad idea, though, assuming
> that bit-packing gives any genuine advantage -- I'm generally skeptical that
> the benefit of being able to squeeze an extra 5% through the network
> outweighs the overhead (especially in code complexity) of bit-twiddling at
> both ends (yes, I realise CPU is more plentiful than bandwidth).  I'd
> personally leave bit twiddling until much later in a project, as I suspect
> it could be an unnecessary and premature optimisation compared to the other
> techniques mentioned.  Bit-twiddling strikes me as an implementation
> optimisation, where algorithmic gains could be much more beneficial... but
> then, I've never written a networked game, so I'm just guessing :)

Networked games are tough, and Derek's concerns are valid.

My point of view on this is that PB cannot handle such a game by itself,
but it defines mechanisms for extensibility that are designed to solve
exactly this sort of problem.  PB's purpose is twofold.  It does provide
a useful, generic, high level protocol that one can use out-of-the-box
in lots of different kinds of applications.  However, I also hope to
provide a sort of meta-standard for protocols that have domain-specific
optimization requirements, so that real systems that solve hard problems
can communicate lower-priority (e.g. "the ones that don't have to be as
close to real time as possible") messages using a mechanism less abusive
to their performance requirements (either on the network in CPU) than
XML-RPC, and easier to add new messages to than something like CORBA. 
For most systems, this should really be good enough.

Ideally I'd like to replace HTTP with it, but that's obviously a bit
more work than has been done :)  (Smash the web!)

Despite the fact that I want to provide a reasonably efficient network
layer, my own humble opinion is that systems that are so
latency-sensitive that PB doesn't work for them out of the box are
probably kidding themselves about the latency issues that arise just by
using the internet... but I think that John Carmack disagrees.

At the very least, PB should help one get off the ground and start
prototyping game functionality really early, before one worries about
seriously packing the network protocol; if one's game is just a 3D
shooter this is probably not that interesting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 232 bytes
Desc: This is a digitally signed message part
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020314/e34e0f76/attachment.pgp