[Twisted-Python] Large file transfers

Glyph Lefkowitz glyph at twistedmatrix.com
Thu Jul 25 15:19:21 EDT 2002


On Thu, 25 Jul 2002 14:45:09 -0400, Steve Waterbury <steve.waterbury at gsfc.nasa.gov> wrote:

> I just noticed item 008 on the twisted TO DO list:
>     File Transfer layer for PB.
> <sophomoric question>
> Would an ftp connection (authenticated using cred, of course) in tandem or
> parallel to the PB connection work?  ... but maybe you are referring to
> implementing file transfer as *part* of the PB protocol, in which case this
> question might not make any sense at all.
> </sophomoric question>

Well, both.  It's more like a convention, really; all the pieces already exist
(serialized remote method calls, waiting until the outgoing buffer is empty) in
order to create a file transfer convention; of course, then you've got all the
usual fun problems; trying to connect with a listening socket on port 0,
backing off and using the existing connection with slightly poorer performance
if one or both of the users are behind a firewall...

> And how close is this to being implemented?

There's a lot of stuff on my plate that comes before it, most of all the 1.0
release.  On the other hand, it's a relatively simple thing to add in.

> My interest is not merely academic -- the application I am working 
> on will be "routinely" transferring 100MB+ files, and I'd like 
> to use the PB as one of our interfaces.

There will be a little bit of overhead in the initial implementation,
considering that it will be using PB calls.  In the short term, the large file
transfers would probably be better handled over HTTP (Twisted's
twisted.web.static.File can easily be used to this end).  FTP is a much nastier
protocol.

> (BTW, one of the components of our app is written in Lisp, so I 
> am thinking about having our Lisp programmer look at Twisted Emacs 
> for hints as to how a Common Lisp implementation of PB could be done ... 
> but I notice that several twisted minions have Lisp in their resumes ... 
> is anyone working on / contemplating a CL implementation of PB?  :^)  

We have contemplated it a lot, but have yet to implement it.  Really the
problem is that "common lisp" isn't a language so much as a bizarre
constellation of languages, none of which have anything to do with each other
when it comes to things like "sockets" and "files".  It hasn't been worth
anybody's effort to go to all that work just for, say, CLisp or SBCL.  AFAIK
it's not even *possible* to do it properly in CLisp yet, due to the absence of
non-blocking sockets, but this could be FUD; haven't read the docs in a while.

However, many of us are fans of Lisp-like languages and I'm sure that more than
one person would be thrilled to see a multi-vendor-supporting CL
implementation of PB.

-- 
 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020725/f1ea464b/attachment.pgp 


More information about the Twisted-Python mailing list