[Twisted-Python] Streaming File Transfer Protocol?

David Bolen db3l.net at gmail.com
Wed Feb 10 16:01:08 EST 2010

Darren Govoni <darren at ontrenet.com> writes:

>>From what I learned in other posts, the dataReceived(self, data): in the
> Echo server
> will get called with out-of-order data/bytes from the client. Of course,
> I could be misinformed,
> but what I understood before was that in this type of Protocol, I would
> have to re-order
> and re-assemble the bytes.  And I'm trying to avoid that, since of
> course, TCP already does it.

Data being received out of order can't happen, as long as the Protocol
is layered on top of TCP, since as you say, TCP already provides that
guarantee.  The dataReceived() method is really just how the data
being received from TCP is handed to the Protocol object.

UDP can be out of order, as it provides very few guarantees above and
beyond IP itself.  But I'm not sure you can layer an IProtocol over
UDP with Twisted.

There is a general issue where you may receive the data in differently
sized chunks in dataReceived() than it might have been transmitted
originally, which is a common source of confusion to people new to
stream protocols, so perhaps you were thinking of that issue?

The stream nature (and possibility for early disconnect from the
client) is why having some internal length information for bulk
transfers is sensible.  For your original question, I was going to
suggest an older posting of mine for a similar situation where I needed
a bulk upload to augment a PB-based server, but it appears that you've
located it in the archives yourself.

> But like I said, I could have been misinformed because it seems pretty
> basic to write 1,2,3
> to a server and have it received 1,2,3, guaranteed.

Yes - TCP guarantees that what you transmit at one end will be
received in order at the other end or not at all (e.g., outages,
disconnects, etc...).  It has a weaker guarantee in terms of no
corruption, but one that is, in combination with typical link layer
protections, generally more than sufficient for the vast majority of
connections using it each day.

-- David

More information about the Twisted-Python mailing list