[Twisted-Python] Streaming File Transfer Protocol?

Darren Govoni darren at ontrenet.com
Sat Feb 13 16:53:38 EST 2010


I spoke too fast. But pardon my noobiness.

Ok, so I am using a simple protocol that is listening on a TCP port.

One the client side, I write 4096 bytes using
self.transport.write(bytes)

on dataReceived side, I get only 1448. 

Now, what I "want" to happen is when I issue a write of a known number
of bytes.
I "want" those bytes to arrive in total because they represent a pickled
object.
The server has no idea if the bytes are split and scattered (again, I
want the
control protocol to take affect).

1) Am I doing something wrong here?
2) Can I force twisted to send ALL the bytes I issue in the write
without
re-thinking TCP or forcing me to re-implement TCP?

thanks!
Darren

On Wed, 2010-02-10 at 18:34 -0500, Darren Govoni wrote:

> Thanks for that explanation David. Makes sense! 
> 
> On Wed, 2010-02-10 at 16:01 -0500, David Bolen wrote: 
> 
> > Darren Govoni <darren at ontrenet.com> writes:
> > 
> > >>From what I learned in other posts, the dataReceived(self, data): in the
> > > Echo server
> > > will get called with out-of-order data/bytes from the client. Of course,
> > > I could be misinformed,
> > > but what I understood before was that in this type of Protocol, I would
> > > have to re-order
> > > and re-assemble the bytes.  And I'm trying to avoid that, since of
> > > course, TCP already does it.
> > 
> > Data being received out of order can't happen, as long as the Protocol
> > is layered on top of TCP, since as you say, TCP already provides that
> > guarantee.  The dataReceived() method is really just how the data
> > being received from TCP is handed to the Protocol object.
> > 
> > UDP can be out of order, as it provides very few guarantees above and
> > beyond IP itself.  But I'm not sure you can layer an IProtocol over
> > UDP with Twisted.
> > 
> > There is a general issue where you may receive the data in differently
> > sized chunks in dataReceived() than it might have been transmitted
> > originally, which is a common source of confusion to people new to
> > stream protocols, so perhaps you were thinking of that issue?
> > 
> > The stream nature (and possibility for early disconnect from the
> > client) is why having some internal length information for bulk
> > transfers is sensible.  For your original question, I was going to
> > suggest an older posting of mine for a similar situation where I needed
> > a bulk upload to augment a PB-based server, but it appears that you've
> > located it in the archives yourself.
> > 
> > > But like I said, I could have been misinformed because it seems pretty
> > > basic to write 1,2,3
> > > to a server and have it received 1,2,3, guaranteed.
> > 
> > Yes - TCP guarantees that what you transmit at one end will be
> > received in order at the other end or not at all (e.g., outages,
> > disconnects, etc...).  It has a weaker guarantee in terms of no
> > corruption, but one that is, in combination with typical link layer
> > protections, generally more than sufficient for the vast majority of
> > connections using it each day.
> > 
> > -- David
> > 
> > 
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20100213/5189171c/attachment.htm 


More information about the Twisted-Python mailing list