[Twisted-Python] FTP without the protocol

David Bolen db3l.net at gmail.com
Fri Oct 25 17:38:01 MDT 2013


Lloyd Carothers <lloyd at passcal.nmt.edu> writes:

> This looks pretty close to what I need and definitely a good starting
> point for me. I can make use of session specific information too,
> which will be nice.
> As you've been using it for a while, have you had any issues. Is it
> robust/stable?

Absolutely robust/stable.  Haven't touched the code since 2007 (well,
aside from one tweak to deal with an application level race condition
under long RTT connections unrelated to the file transfer itself).
Looks like there have been around 60000 media file transfers totaling
about 650GB of data so far.

> Would you be willing to share this as well?

It's part of a larger system that I can't really share in total, but I
could probably snip an extract of the PB part related to file I/O.
It's not really anything special - just a typical PB published object
with some remote methods.  You'd have to be willing to deal with it in
an "as-is" state though.  If interested, drop me a note directly.

> Gotcha.
> Maybe this is new since you wrote the above, but is FileSender the
> producer to use here?

That's the one I use on the client side, yes.  On the receiver the raw
data receipt is just part of the server side protocol definition.

> Perhaps its not fully developed but should there not also be a
> compliment FileReceiver  to consume the file and write it out?

Maybe.  I don't think such a helper exists in the version of Twisted I
was using, but it may also have been simpler enough that I didn't look too
hard (plus I wanted a running crc calculation).

I think it's also a little trickier on the receiving side than
sending.  On the sending side it's easier to delegate transmission of
the data, while on the receiving side the data (and connection status)
occurs as part of your protocol handler.  So at the least something
like FileReceiver would probably need to be a mix-in (taking over
dataReceived in some way) rather than an independent function.

The receiving side is also fairly trivial - in my case, it's not much
more than the few lines in dataReceived.

> Generally FTP works ok but some organizations firewalls do strange
> things with that traffic, and even the good connections seem to have
> sporadic drops which often aren't handled well, at least with proftp.

Yeah, it's an ugly network world out there :-)

Note that with an FTP replacement, you'll still be responsible for
handling disconnects and/or resuming uploads.  In my current system
all uploads go to a dedicated upload folder by job, which is moved to
its final location during the job commit, or completely discarded if
something goes wrong.  But I have the luxury of having tight control
of the client (who is an audio post-production house publishing jobs
for clients to review), so failures between the client and the server
in my case are actually reasonably rare.

-- David





More information about the Twisted-Python mailing list