[Twisted-Python] Conch SFTP Questions
Robert DiFalco
robert.difalco at gmail.com
Tue Sep 22 16:47:14 MDT 2020
Thanks! That is the full code. `connect` is from the conch library.
On Tue, Sep 22, 2020 at 12:57 PM Adi Roiban <adi at roiban.ro> wrote:
> Hi Robers
>
> On Tue, 22 Sep 2020 at 16:43, Robert DiFalco <robert.difalco at gmail.com>
> wrote:
>
>> Hey folks, I've cobbled together an SFTP client based on bits and pieces
>> I've found around the web. The issue is that it appears to be almost one
>> shot. I will need to send many files (the number not known ahead of time).
>> It's not clear to me when the connection is closed or how many factories
>> I'm creating. All the code I've grabbed looks like it's creating a new
>> factory for every SFTP file I send. Here's some of the code I have. It's
>> fairly straight forward in that it creates a directory if it doesn't exist
>> and then writes a file.
>>
>> @attr.s(frozen=True)
>> class FileInfo(object):
>> """
>> Class that tells SFTP details about the file to send.
>> """
>> directory = attr.ib(converter=str) # type: str
>> name = attr.ib(converter=str) # type: str
>> data = attr.ib() # type: str
>> chunk_size = attr.ib(converter=int, default=CHUNK_SIZE) # type: int
>>
>> def to_path(self):
>> """
>> Turns the folder and file name into a file path.
>> """
>> return self.directory + "/" + self.name
>>
>>
>> @attr.s(frozen=True)
>> class SFTPClientOptions(object):
>> """
>> Client options for sending SFTP files.
>>
>> :param host: the host of the SFTP server
>> :param port: the port ofo the SFTP server
>> :param fingerprint: the expected fingerprint of the host
>> :param user: the user to login as
>> :param identity: the identity file, optional and like the "-i" command line option
>> :param password: an optional password
>> """
>> host = attr.ib(converter=str) # type: str
>> port = attr.ib(converter=int) # type: int
>> fingerprint = attr.ib(converter=str) # type: str
>> user = attr.ib(converter=str) # type: str
>> identity = attr.ib(converter=optional(str), default=None) # type: Optional[str]
>> password = attr.ib(converter=optional(str), default=None) # type: Optional[str]
>>
>>
>> @inlineCallbacks
>> def sftp_send(client_options, file_info):
>> # type: (SFTPClientOptions, FileInfo)->Deferred
>> """
>> Primary function to send an file over SFTP. You can send a password, identity, or both.
>> :param client_options: the client connection options
>> :param file_info: contains the file info to write
>> :return: A deferred that signals "OK" if successful.
>> """
>> options = ClientOptions()
>> options["host"] = client_options.host
>> options["port"] = client_options.port
>> options["password"] = client_options.password
>> options["fingerprint"] = client_options.fingerprint
>>
>> if client_options.identity:
>> options.identitys = [client_options.identity]
>>
>> conn = SFTPConnection()
>> auth = SFTPUserAuthClient(client_options.user, options, conn)
>> yield connect(client_options.host, client_options.port, options, _verify_host_key, auth)
>>
>> sftpClient = yield conn.getSftpClientDeferred()
>> yield _send_file(sftpClient, file_info)
>>
>> returnValue("OK")
>>
>>
>> def _verify_host_key(transport, host, pubKey, fingerprint):
>> """
>> Verify a host's key. Based on what is specified in options.
>>
>> @param host: Due to a bug in L{SSHClientTransport.verifyHostKey}, this is
>> always the dotted-quad IP address of the host being connected to.
>> @type host: L{str}
>>
>> @param transport: the client transport which is attempting to connect to
>> the given host.
>> @type transport: L{SSHClientTransport}
>>
>> @param fingerprint: the fingerprint of the given public key, in
>> xx:xx:xx:... format.
>>
>> @param pubKey: The public key of the server being connected to.
>> @type pubKey: L{str}
>>
>> @return: a L{Deferred} which is success or error
>> """
>> expected = transport.factory.options.get("fingerprint", "no_fingerprint")
>> if fingerprint == expected:
>> return succeed(1)
>>
>> log.error(
>> "SSH Host Key fingerprint of ({fp}) does not match the expected value of ({expected}).",
>> fp=fingerprint, expected=expected)
>>
>> return fail(ConchError("Host fingerprint is unexpected."))
>>
>>
>> class SFTPSession(SSHChannel):
>> """
>> Creates an SFTP session.
>> """
>> name = "session"
>>
>> @inlineCallbacks
>> def channelOpen(self, whatever):
>> """
>> Called when the channel is opened. "whatever" is any data that the
>> other side sent us when opening the channel.
>>
>> @type whatever: L{bytes}
>> """
>> yield self.conn.sendRequest(self, "subsystem", NS("sftp"), wantReply=True)
>>
>> client = FileTransferClient()
>> client.makeConnection(self)
>> self.dataReceived = client.dataReceived
>> self.conn.notifyClientIsReady(client)
>>
>>
>> class SFTPConnection(SSHConnection):
>> def __init__(self):
>> """
>> Adds a deferred here so client can add a callback when the SFTP client is ready.
>> """
>> SSHConnection.__init__(self)
>> self._sftpClient = Deferred()
>>
>> def serviceStarted(self):
>> """
>> Opens an SFTP session when the SSH connection has been started.
>> """
>> self.openChannel(SFTPSession())
>>
>> def notifyClientIsReady(self, client):
>> """
>> Trigger callbacks associated with our SFTP client deferred. It's ready!
>> """
>> self._sftpClient.callback(client)
>>
>> def getSftpClientDeferred(self):
>> return self._sftpClient
>>
>>
>> class SFTPUserAuthClient(SSHUserAuthClient):
>> """
>> Twisted Conch doesn't have a way of getting a password. By default it gets it from stdin. This allows it
>> to be retrieved from options instead.
>> """
>> def getPassword(self, prompt = None):
>> """
>> Get the password from the client options, is specified.
>> """
>> if "password" in self.options:
>> return succeed(self.options["password"])
>>
>> return SSHUserAuthClient.getPassword(self, prompt)
>>
>>
>> @inlineCallbacks
>> def _send_file(client, file_info):
>> # type: (FileTransferClient, FileInfo) -> Deferred
>> """
>> Creates a directory if required and then creates the file.
>> :param client: the SFTP client to use
>> :param file_info: contains file name, directory, and data
>> """
>> try:
>> yield client.makeDirectory(file_info.directory, {})
>>
>> except SFTPError as e:
>> # In testing on various system, either a 4 or an 11 will indicate the directory
>> # already exist. We are fine with that and want to continue if it does. If we misinterpreted
>> # error code here we are probably still ok since we will just get the more systemic error
>> # again on the next call to openFile.
>> if e.code != 4 and e.code != 11:
>> raise e
>>
>> f = yield client.openFile(file_info.to_path(), FXF_WRITE | FXF_CREAT | FXF_TRUNC, {})
>>
>> try:
>> yield _write_chunks(f, file_info.data, file_info.chunk_size)
>>
>> finally:
>> yield f.close()
>>
>>
>> @inlineCallbacks
>> def _write_chunks(f, data, chunk_size):
>> # type: (ClientFile, str, int) -> Deferred
>> """
>> Convenience function to write data in chunks
>>
>> :param f: the file to write to
>> :param data: the data to write
>> :param chunk_size: the chunk size
>> """
>> for offset in range(0, len(data), chunk_size):
>> chunk = data[offset: offset + chunk_size]
>> yield f.writeChunk(offset, chunk)
>>
>>
>> It gets called like this:
>>
>> return sftp.sftp_send(
>> client_options=SFTPClientOptions(
>> host=self.options.host,
>> port=self.options.port,
>> user=self.options.user,
>> fingerprint=self.options.fingerprint,
>> identity=getattr(self.options, "identity", None),
>> password=self._getPassword()),
>> file_info=sftp.FileInfo(
>> directory=self.options.directory,
>> name=fileName,
>> data=data,
>> chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE)))
>>
>> But I supposed I'd like to see something more like this:
>>
>> sftpClient = self.getSftpClient(
>> client_options=SFTPClientOptions(
>> host=self.options.host,
>> port=self.options.port,
>> user=self.options.user,
>> fingerprint=self.options.fingerprint,
>> identity=getattr(self.options, "identity", None),
>> password=self._getPassword()))
>>
>> return sftpClient.send(
>> file_info=sftp.FileInfo(
>> directory=self.options.directory,
>> name=fileName,
>> data=data,
>> chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE)))
>>
>> Where sftpClient reuses the existing SSH connection if it is active
>> (rather than logging in each time). But maybe the sftp service doesn't
>> multiplex so I have to create a new SSHClientFactory every time I want to
>> send a distinct file?
>>
>> Sorry for all the questions, new to twisted and a bit confused. Thanks!
>>
>> Robert
>>
>> It would help to have the full code...maybe a gist or repo.
> I am not sure what `connect` from `yield connect(client_options.host,
> client_options.port, options, _verify_host_key, auth)` is.
>
> You will need to understand the low-level Twisted connection API and
> implement a reconnecting factory.
>
> When a new client-side connection is made, Twisted will use a factory to
> create the protocol/code used to handle that connection,
> You will then need to hook into the connectionLost method and do an
> auto-connection if the connection is lost (when you were not expecting it).
>
> ---------
>
> For my project, I am doing in this way:
>
> I have my own subclass of FileTransferClient which overwrites the default
> FileTransferClient,connectionLost method.
> With that, I am notified when the SFTP subsystem was closed and I can then
> trigger a new connection
>
> -------------
>
> If you want to reuse an SFTP session for multiple operations just reuse
> the `sftpClient` instance that you got to trigger multiple operations
>
> sftpClient = yield conn.getSftpClientDeferred()
> for file_info in list_of_files_to_send:
> yield _send_file(sftpClient, file_info)
>
> ----------
>
> Hope it helps
>
> --
> Adi Roiban
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200922/04ce963d/attachment-0001.htm>
More information about the Twisted-Python
mailing list