[Twisted-Python] Large file transfers

Steve Waterbury steve.waterbury at gsfc.nasa.gov
Fri Jul 26 17:49:04 EDT 2002


Glyph Lefkowitz wrote:

> As several people have pointed out to me, sometimes this additional overhead is
> worth paying in order to get rid of the TCP connection startup cost.  Of
> course, this will vary with your application, but it's probably worthwhile to
> have the ability to reuse the connection.
> 
> In addition, sometimes you want to communicate large *objects*, and not just
> large *files*; in those cases, it's really handy to have objects which can be
> "paged" between systems, interleaved with other messages.

Yes, both of these are important considerations for our app.  

> This is also a decent basis for "file transfer" between two objects, since
> "very large string, stored on disk" is a degenerate case of "very large object,
> stored somewhere".  I've checked in an implementation of this to
> twisted.spread.util.Pager.  Not much in the way of docs yet, but the test cases
> (twisted.test.test_pb, look for "pager") should at least explain some of it.

Great -- I'll take a look at that.  Thanks!  

> > [SW wrote:]...  Let us know if other vendor
> > or open-source Lisps are of interest, etc.
> 
> I can speak only to my own interests.  Other members of the Twisted community
> are working in radically different environments and on radically different
> problems.  As long as you asked, though ...
> 
> While I appreciate that they're technically superior, the proprietary lisps
> don't interest me; the bang for your buck of using a proprietary language
> solution is seldom worth the hefty license fees (especially in a situations
> where "interoperability" and "ubiquity" are the most interesting parts, like
> most of the applications I've come up with for Twisted).

Understood.  Since we have limited manpower for our Lisp work, we tend 
to value the power of commercial development tools, but we do care 
about portability, so we try to adhere to ANSI CL -- see:
http://exp-engine.sourceforge.net/lisp.html
(And it is *very* important that there are no *run-time* license fees!
Anyone can download our stuff and run it.)

Express Engine, our Lisp project, is somewhat esoteric, so even though 
it's on SF no other developers have joined. ;^)  If anyone is curious, 
see:  http://exp-engine.sourceforge.net 

Express Engine encapsulates some very important functions for our app, 
enabling us to munge data produced by various CAD/CAE/CAM ("CAX") 
tools, for which the most widely implemented exchange standard is 
ISO 10303 (STEP).  Express Engine encapsulates our STEP data I/O and 
mapping functionality.  (Like I said, somewhat esoteric! :^)  As you 
can imagine, STEP files produced by CAX tools can be quite huge.  

This also hints at why Twisted is such a natural fit for our app:  a 
multi-player networked game environment is not all that different 
from a massive collaborative engineering environment ... at the 
infrastructure level, essentially identical.

Cheers,
-- Steve.

Stephen C. Waterbury  http://misspiggy.gsfc.nasa.gov/people/waterbug.html




More information about the Twisted-Python mailing list