[Twisted-Python] Conch SFTP Questions
Adi Roiban
adi at roiban.ro
Tue Sep 22 13:56:37 MDT 2020
Hi Robers
On Tue, 22 Sep 2020 at 16:43, Robert DiFalco <robert.difalco at gmail.com>
wrote:
> Hey folks, I've cobbled together an SFTP client based on bits and pieces
> I've found around the web. The issue is that it appears to be almost one
> shot. I will need to send many files (the number not known ahead of time).
> It's not clear to me when the connection is closed or how many factories
> I'm creating. All the code I've grabbed looks like it's creating a new
> factory for every SFTP file I send. Here's some of the code I have. It's
> fairly straight forward in that it creates a directory if it doesn't exist
> and then writes a file.
>
> @attr.s(frozen=True)
> class FileInfo(object):
> """
> Class that tells SFTP details about the file to send.
> """
> directory = attr.ib(converter=str) # type: str
> name = attr.ib(converter=str) # type: str
> data = attr.ib() # type: str
> chunk_size = attr.ib(converter=int, default=CHUNK_SIZE) # type: int
>
> def to_path(self):
> """
> Turns the folder and file name into a file path.
> """
> return self.directory + "/" + self.name
>
>
> @attr.s(frozen=True)
> class SFTPClientOptions(object):
> """
> Client options for sending SFTP files.
>
> :param host: the host of the SFTP server
> :param port: the port ofo the SFTP server
> :param fingerprint: the expected fingerprint of the host
> :param user: the user to login as
> :param identity: the identity file, optional and like the "-i" command line option
> :param password: an optional password
> """
> host = attr.ib(converter=str) # type: str
> port = attr.ib(converter=int) # type: int
> fingerprint = attr.ib(converter=str) # type: str
> user = attr.ib(converter=str) # type: str
> identity = attr.ib(converter=optional(str), default=None) # type: Optional[str]
> password = attr.ib(converter=optional(str), default=None) # type: Optional[str]
>
>
> @inlineCallbacks
> def sftp_send(client_options, file_info):
> # type: (SFTPClientOptions, FileInfo)->Deferred
> """
> Primary function to send an file over SFTP. You can send a password, identity, or both.
> :param client_options: the client connection options
> :param file_info: contains the file info to write
> :return: A deferred that signals "OK" if successful.
> """
> options = ClientOptions()
> options["host"] = client_options.host
> options["port"] = client_options.port
> options["password"] = client_options.password
> options["fingerprint"] = client_options.fingerprint
>
> if client_options.identity:
> options.identitys = [client_options.identity]
>
> conn = SFTPConnection()
> auth = SFTPUserAuthClient(client_options.user, options, conn)
> yield connect(client_options.host, client_options.port, options, _verify_host_key, auth)
>
> sftpClient = yield conn.getSftpClientDeferred()
> yield _send_file(sftpClient, file_info)
>
> returnValue("OK")
>
>
> def _verify_host_key(transport, host, pubKey, fingerprint):
> """
> Verify a host's key. Based on what is specified in options.
>
> @param host: Due to a bug in L{SSHClientTransport.verifyHostKey}, this is
> always the dotted-quad IP address of the host being connected to.
> @type host: L{str}
>
> @param transport: the client transport which is attempting to connect to
> the given host.
> @type transport: L{SSHClientTransport}
>
> @param fingerprint: the fingerprint of the given public key, in
> xx:xx:xx:... format.
>
> @param pubKey: The public key of the server being connected to.
> @type pubKey: L{str}
>
> @return: a L{Deferred} which is success or error
> """
> expected = transport.factory.options.get("fingerprint", "no_fingerprint")
> if fingerprint == expected:
> return succeed(1)
>
> log.error(
> "SSH Host Key fingerprint of ({fp}) does not match the expected value of ({expected}).",
> fp=fingerprint, expected=expected)
>
> return fail(ConchError("Host fingerprint is unexpected."))
>
>
> class SFTPSession(SSHChannel):
> """
> Creates an SFTP session.
> """
> name = "session"
>
> @inlineCallbacks
> def channelOpen(self, whatever):
> """
> Called when the channel is opened. "whatever" is any data that the
> other side sent us when opening the channel.
>
> @type whatever: L{bytes}
> """
> yield self.conn.sendRequest(self, "subsystem", NS("sftp"), wantReply=True)
>
> client = FileTransferClient()
> client.makeConnection(self)
> self.dataReceived = client.dataReceived
> self.conn.notifyClientIsReady(client)
>
>
> class SFTPConnection(SSHConnection):
> def __init__(self):
> """
> Adds a deferred here so client can add a callback when the SFTP client is ready.
> """
> SSHConnection.__init__(self)
> self._sftpClient = Deferred()
>
> def serviceStarted(self):
> """
> Opens an SFTP session when the SSH connection has been started.
> """
> self.openChannel(SFTPSession())
>
> def notifyClientIsReady(self, client):
> """
> Trigger callbacks associated with our SFTP client deferred. It's ready!
> """
> self._sftpClient.callback(client)
>
> def getSftpClientDeferred(self):
> return self._sftpClient
>
>
> class SFTPUserAuthClient(SSHUserAuthClient):
> """
> Twisted Conch doesn't have a way of getting a password. By default it gets it from stdin. This allows it
> to be retrieved from options instead.
> """
> def getPassword(self, prompt = None):
> """
> Get the password from the client options, is specified.
> """
> if "password" in self.options:
> return succeed(self.options["password"])
>
> return SSHUserAuthClient.getPassword(self, prompt)
>
>
> @inlineCallbacks
> def _send_file(client, file_info):
> # type: (FileTransferClient, FileInfo) -> Deferred
> """
> Creates a directory if required and then creates the file.
> :param client: the SFTP client to use
> :param file_info: contains file name, directory, and data
> """
> try:
> yield client.makeDirectory(file_info.directory, {})
>
> except SFTPError as e:
> # In testing on various system, either a 4 or an 11 will indicate the directory
> # already exist. We are fine with that and want to continue if it does. If we misinterpreted
> # error code here we are probably still ok since we will just get the more systemic error
> # again on the next call to openFile.
> if e.code != 4 and e.code != 11:
> raise e
>
> f = yield client.openFile(file_info.to_path(), FXF_WRITE | FXF_CREAT | FXF_TRUNC, {})
>
> try:
> yield _write_chunks(f, file_info.data, file_info.chunk_size)
>
> finally:
> yield f.close()
>
>
> @inlineCallbacks
> def _write_chunks(f, data, chunk_size):
> # type: (ClientFile, str, int) -> Deferred
> """
> Convenience function to write data in chunks
>
> :param f: the file to write to
> :param data: the data to write
> :param chunk_size: the chunk size
> """
> for offset in range(0, len(data), chunk_size):
> chunk = data[offset: offset + chunk_size]
> yield f.writeChunk(offset, chunk)
>
>
> It gets called like this:
>
> return sftp.sftp_send(
> client_options=SFTPClientOptions(
> host=self.options.host,
> port=self.options.port,
> user=self.options.user,
> fingerprint=self.options.fingerprint,
> identity=getattr(self.options, "identity", None),
> password=self._getPassword()),
> file_info=sftp.FileInfo(
> directory=self.options.directory,
> name=fileName,
> data=data,
> chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE)))
>
> But I supposed I'd like to see something more like this:
>
> sftpClient = self.getSftpClient(
> client_options=SFTPClientOptions(
> host=self.options.host,
> port=self.options.port,
> user=self.options.user,
> fingerprint=self.options.fingerprint,
> identity=getattr(self.options, "identity", None),
> password=self._getPassword()))
>
> return sftpClient.send(
> file_info=sftp.FileInfo(
> directory=self.options.directory,
> name=fileName,
> data=data,
> chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE)))
>
> Where sftpClient reuses the existing SSH connection if it is active
> (rather than logging in each time). But maybe the sftp service doesn't
> multiplex so I have to create a new SSHClientFactory every time I want to
> send a distinct file?
>
> Sorry for all the questions, new to twisted and a bit confused. Thanks!
>
> Robert
>
> It would help to have the full code...maybe a gist or repo.
I am not sure what `connect` from `yield connect(client_options.host,
client_options.port, options, _verify_host_key, auth)` is.
You will need to understand the low-level Twisted connection API and
implement a reconnecting factory.
When a new client-side connection is made, Twisted will use a factory to
create the protocol/code used to handle that connection,
You will then need to hook into the connectionLost method and do an
auto-connection if the connection is lost (when you were not expecting it).
---------
For my project, I am doing in this way:
I have my own subclass of FileTransferClient which overwrites the default
FileTransferClient,connectionLost method.
With that, I am notified when the SFTP subsystem was closed and I can then
trigger a new connection
-------------
If you want to reuse an SFTP session for multiple operations just reuse the
`sftpClient` instance that you got to trigger multiple operations
sftpClient = yield conn.getSftpClientDeferred()
for file_info in list_of_files_to_send:
yield _send_file(sftpClient, file_info)
----------
Hope it helps
--
Adi Roiban
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200922/95ca83f5/attachment-0001.htm>
More information about the Twisted-Python
mailing list