[Twisted-Python] Retry: Conch Issues

[Robert DiFalco]

Thank you so much Aldi! Yes there are so many little decoupled pieces it's
difficult to figre out how to use them in concert. .Let's leave aside the
reconnecting option for a minute and just talk about how to percolate
errors up. Because currently I want to be able to always return from a
FileTransferClient call. By upleveling things do you mean like this?

Wait, ha, I actually don't know because the deferreds are one shot? Why
exactly is makeDirectory not firing any callbacks or errbacks and what can
I add at the Factory level to make it do so?

class MySSHClientFactory(SSHClientFactory):
    def clientConnectionLost(self, connector, reason):
        self.op.addErrback(reason)

    def makeDirectory(self, path, attrs):
        self.op = Deferred()
        def _cbSuccess(result):
            self.op.callback(result)

        return self.sftpClient.makeDirectory(path,
attrs).addCallback(_cbSuccess)d

That won't work. If you can give me a breadcrumb or idiom I can apply it
across everything. I'm just too new to twisted to know the correct idiom
here. Totally good with moving the methods up to the factory, just not sure
how.



On Thu, Sep 24, 2020 at 5:36 PM Adi Roiban <adi at roiban.ro> wrote:

> Hi
>
> On Fri, 25 Sep 2020 at 00:15, Robert DiFalco <robert.difalco at gmail.com>
> wrote:
>
>> I am having troubles with Conch. I've been able to reproduce them in a
>> unit test and all the code is here along with requirements.txt:
>>    https://github.com/radifalco/conch_issues
>>
>> I'm trying to reuse my connection, but if it is idle too long the server
>> shuts it down. What I don't understand is why the next call to
>> `FileTransferClient` does not call back or errback? It will just hang
>> forever even though the Factory and Transport know the connection was
>> closed (as evidenced by the log messages).
>>
>> I don't ever want to hang, if I use FileTransferClient I would like it
>> always call a callback or errback. I feel like Conch is missing a deferred
>> somewhere but damned if I can find it.
>>
>> Thanks!
>>
>> Have you checked
> https://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
> ?
>
> I see that `connector.connect()` is commented in your code.
>
> The theory is that once a client connection is lost, you will need to call
> again the whole chain starting with reactor.connectTCP.
>
> ---------
>
> I know the pain here with SFTP as there is a lot of wrapping: TCP client,
> SSH high level protocol, SSH session, SSH channel, SSH subsystem.
>
> One option is to use MySSHClientFactory as a proxy for any SFTP operation.
> When the SSH channel is open and you have received the response from the
> subsystem request, pass the SFTP client all the way down to the factory.
> The factory will know when the connection is lost and then trigger a new
> connection... which will update the SFTP client instance on the factory.
>
> That is, instead of getting the SFTP client, the SFTP client will be
> pushed to the factory.
>
> I think that the main issue with your code is that it tries to work
> directly with the SFTP client, instead of using the factory as a proxy for
> SFTP operations.
>
> A protocol represents a single connection that is already made and it
> can't reconnect itself.
> The factory is responsible for handling new connections, or triggering new
> connections.
>
> So, your high-level API is using a very low-level object.
> The SFTP client represents an SSH connection that was already made,
> authenticated and the SFTP subsystem requested.
>
> For my production code I am doing something like this:
> MySSHClientFactory is instantiated with the reactor and the
> SFTPClientOptions.
> Then, when I want to list a directory I call
> sftp_factory.listDir('some/path')
> The factory can then check if it needs to get an SFTP client or reuse an
> existing one.
> A SFTP client is obtained by triggering a new connectTCP to the reactor
> with `self` as the factory.
>
> Does it make sense?
>
> --
> Adi Roiban
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200924/ce0015e7/attachment.htm>