[Twisted-Python] troubles with FTP server and async file writing

[Jean-Paul Calderone]

On Fri, 3 Oct 2008 18:26:18 -0700, Brian Warner <warner at lothar.com> wrote:
>[snip]
>The specific problem is in the way that
>twisted.protocols.ftp.FTP.ftp_STOR() interacts with the IConsumer that it
>gets.. the sequence goes like this:
>
> [snip]
> 6: the DTP connection is closed, _onConnLost is fired
> 7:  DTP._unregConsumer sends unregisterProducer(), and ignores the result
> 8:  FTP.cbConsumer()'s chain regains control, runs cbSent (via Deferred)
> 9:  cbSent returns 226 Transfer Complete
> [snip]
>
>Most of these steps use Deferreds, so the opening of the file can take as
>long as we need, but since DTP._unregConsumer ignores the result of
>consumer.unregisterProducer in step #7, there's no way for the backend to
>stall the delivery of the 226 until the file has actually been uploaded.
>
>We need something to address this, because FTP clients are correctly assuming
>that once they get the 226 Transfer Complete, the file will be in place.
>

Yep.  You're right that the FTP server protocol implementation currently
assumes that once it has written the last chunk to the consumer that the
file is successfully uploaded.  It would be a good enhancement to server
to allow application code to insert one more Deferred into processing of
uploads after the last write and before the success notification.

Unfortunately, there's no way to signal this with `IConsumer`.  Likewise
(as you pointed out) `IFinishableConsumer.finish` doesn't have very well
defined semantics.  In particular, there's no way to infer from its docs
that a Deferred might be returned.

>
>I guess Plan C would be to involve the IWriteFile instance, adding a finish()
>method of some sort, making this an FTP-specific solution.
>
>So, any thoughts? Something like this just at the VFS level would allow
>backends to use asynchronous-writes, which feels like a significant gap in
>the current functionality. It might be better to address it at the IConsumer
>level, but that would touch more (and better-established) code.
>

Trying to solve this in VFS before it has been solved for any specific use
case would be premature.  Similarly, making a change to `IConsumer` before
implementing the idea even once should be avoided.  I would suggest adding
a new (optional) interface to the FTP server implementation which supports
the close hook that's needed here.  Once something is working, then we can
try to generalize it.

Jean-Paul