[Twisted-Python] Serializing cftp and ssh sessions

[exarkun at twistedmatrix.com]

On 02:05 pm, axel.rau at chaos1.de wrote:
>
>Am 19.10.2014 um 22:44 schrieb exarkun at twistedmatrix.com:
>>On 17 Oct, 08:50 am, axel.rau at chaos1.de wrote:
>>>
>>>Am 14.10.2014 um 17:07 schrieb Axel Rau <Axel.Rau at chaos1.de>:
>>>>>We might need more to go on.  Do you have a complete, minimal 
>>>>>example? (See http://sscce.org/)
>>>>The attached zip archive
>>>Did someone could spend the time to look at it?
>>>Is my goal utopian?
>>
>>I can't tell if this represents a bug in Twisted or not.  After a 
>>quick look through the code, I’m not really clear on why the program 
>>would ever exit (I didn’t find the calls to `reactor.stop`, I guess 
>>they’re hidden somewhere in conch „library“ code),
>I must prevent this stopping of the reactor, because I’m unable to 
>start it again.
>I’m always getting twisted.internet.error.ReactorNotRestartable after 
>trying it while reactor.running is False.

Okay.  I guess I misunderstood part of the problem statement.  I thought 
you wanted it to stop at some point.
>>  nor quite which code paths you expect to work and which you expect to 
>>fail.
>>
>>
>>What is the expected behavior if I run test.py after editing it to 
>>point at a host I have access to?
>If you put a reactor.stop() at the end of ssh._ebExit, test.py should 
>run to completion after doing the remote file operations and executing 
>the remote commands.
>>
>>Is that the desired behavior?
>No.
>I need a solution where the call to executeRemoteCommands() is delayed 
>until all remote file operations are complete. Similar the next call to 
>distributeFiles (with a different host) must wait until the previous 
>call to executeRemoteCommands has no outstanding operations.
>>
>>If there is a second case, what one edit can I make to observe that 
>>behavior?
>No.
>>  How does the behavior differ from what you’re trying to achieve?
>See above. I must synchronize operations at 2 points in a loop.
>I tried to do this with callbacks, called at connection shutdown.
>What do you think about this?

That sounds like the correct solution to me.  The simplest way to 
structure your code is to make the APIs for starting these two 
operations return a Deferred that fires when the operation is complete. 
Then you can use these Deferreds to construct the desired control flow 
for starting the operations.
>>
>>Separately:
>>
>>* You should rewrite this code to not use 
>>twisted.conch.client.connect. That's some random library code no one 
>>ever expected application code to use.  It's public, so you can use it 
>>if you want, but that's no promise it's any good.
>>
>>* Use twisted.conch.endpoints.SSHCommandClientEndpoint to perform 
>>`exec` requests
>>
>>* Consider pitching in to complete 
>>https://twistedmatrix.com/trac/ticket/6617 so you can use the new 
>>`SSHSubsystemClientEndpoint` API that ticket introduces to set up your 
>>SFTP connections.
>I looked at endpoints before, but failed to implement public key 
>authentication and a callback when complete.

You should be able to do key-based authentication with the endpoint. 
SSHCommandClientEndpoint.newConnection accepts a `keys` argument if you 
want to specify the keys to use explicitly.  It also accepts an 
`agentEndpoint` argument if you instead want to point it at an SSH key 
agent which will perform key operations for it.

Jean-Paul