[Twisted-Python] Streaming File Transfer Protocol?
darren at ontrenet.com
Sat Feb 13 16:53:38 EST 2010
I spoke too fast. But pardon my noobiness.
Ok, so I am using a simple protocol that is listening on a TCP port.
One the client side, I write 4096 bytes using
on dataReceived side, I get only 1448.
Now, what I "want" to happen is when I issue a write of a known number
I "want" those bytes to arrive in total because they represent a pickled
The server has no idea if the bytes are split and scattered (again, I
control protocol to take affect).
1) Am I doing something wrong here?
2) Can I force twisted to send ALL the bytes I issue in the write
re-thinking TCP or forcing me to re-implement TCP?
On Wed, 2010-02-10 at 18:34 -0500, Darren Govoni wrote:
> Thanks for that explanation David. Makes sense!
> On Wed, 2010-02-10 at 16:01 -0500, David Bolen wrote:
> > Darren Govoni <darren at ontrenet.com> writes:
> > >>From what I learned in other posts, the dataReceived(self, data): in the
> > > Echo server
> > > will get called with out-of-order data/bytes from the client. Of course,
> > > I could be misinformed,
> > > but what I understood before was that in this type of Protocol, I would
> > > have to re-order
> > > and re-assemble the bytes. And I'm trying to avoid that, since of
> > > course, TCP already does it.
> > Data being received out of order can't happen, as long as the Protocol
> > is layered on top of TCP, since as you say, TCP already provides that
> > guarantee. The dataReceived() method is really just how the data
> > being received from TCP is handed to the Protocol object.
> > UDP can be out of order, as it provides very few guarantees above and
> > beyond IP itself. But I'm not sure you can layer an IProtocol over
> > UDP with Twisted.
> > There is a general issue where you may receive the data in differently
> > sized chunks in dataReceived() than it might have been transmitted
> > originally, which is a common source of confusion to people new to
> > stream protocols, so perhaps you were thinking of that issue?
> > The stream nature (and possibility for early disconnect from the
> > client) is why having some internal length information for bulk
> > transfers is sensible. For your original question, I was going to
> > suggest an older posting of mine for a similar situation where I needed
> > a bulk upload to augment a PB-based server, but it appears that you've
> > located it in the archives yourself.
> > > But like I said, I could have been misinformed because it seems pretty
> > > basic to write 1,2,3
> > > to a server and have it received 1,2,3, guaranteed.
> > Yes - TCP guarantees that what you transmit at one end will be
> > received in order at the other end or not at all (e.g., outages,
> > disconnects, etc...). It has a weaker guarantee in terms of no
> > corruption, but one that is, in combination with typical link layer
> > protections, generally more than sufficient for the vast majority of
> > connections using it each day.
> > -- David
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python