[Twisted-Python] Streaming File Transfer Protocol?

Andrew Bennetts andrew at bemusement.org
Sat Feb 13 18:06:19 EST 2010

Darren Govoni wrote:
>    I spoke too fast. But pardon my noobiness.
>    Ok, so I am using a simple protocol that is listening on a TCP port.
>    One the client side, I write 4096 bytes using self.transport.write(bytes)
>    on dataReceived side, I get only 1448.
>    Now, what I "want" to happen is when I issue a write of a known
>    number of bytes.  I "want" those bytes to arrive in total because
>    they represent a pickled object.  The server has no idea if the
>    bytes are split and scattered (again, I want the control protocol
>    to take affect).
>    1) Am I doing something wrong here?

Yes.  You are expecting guarantees that aren't provided by TCP.  TCP
only guarantees that a stream of bytes will arrive in order (or not at
all).  It guarantees nothing about what size packets those bytes will
arrive in.  As you've already noticed it is common for packets to be
broken up into less than 1500 bytes (1500 is a common MTU for internet
hops[1]), and depending on the network links between your hosts it could
quite easily be broken up even further.

So, you need to use some sort of framing in your protocol.

>    2) Can I force twisted to send ALL the bytes I issue in the write without
>    re-thinking TCP or forcing me to re-implement TCP?

Twisted by itself cannot solve this for you.  Even if there were some
way to guarantee that your write was written to the network in a single
packet (and there can't be due to how TCP works, but let's assume you
can), it wouldn't matter because the next hop in the network connection
would be free to break up that large packet into smaller pieces, and
probably would.

Your data needs to be self-describe how long each message is, i.e. frame
them.  See <http://www.xml.com/pub/a/ws/2003/11/25/protocols.html> (and
the rest of the series at <http://www.xml.com/pub/au/215>).

Twisted provides some facilities for building protocols with different
kinds of framing, e.g. delimited by a special marker (LineReceiver) or
length-prefixed (NetstringReceiver, Int32StringReceiver).  Those classes
can be found in twisted.protocols.basic.


[1] http://en.wikipedia.org/wiki/Maximum_transmission_unit

More information about the Twisted-Python mailing list