Thank you very much! :)<br><br><div><span class="gmail_quote">On 7/12/07, <b class="gmail_sendername">David Bolen</b> <<a href="mailto:db3l.net@gmail.com">db3l.net@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
"steven wang" <<a href="mailto:steven.zdwang@gmail.com">steven.zdwang@gmail.com</a>> writes:<br><br>> But I want to receive binary data in my protocol.<br><br>Even if you start with a non-binary header, you can switch to
<br>receiving binary information at any time by going using the raw mode<br>of most of the basic protocols. And having some sort of ASCII header<br>prior to the raw data is often a very simple way to handle things<br>(something in common with a tremendous number of standard TCP-based
<br>protocols).<br><br>Your original post had a fairly straight-forward ASCII header that I<br>think would probably be fine. What you're probably missing is the<br>concept of switching to a raw binary receive mode which then switches
<br>your protocol from getting data in its lineReceived method to having<br>rawDataReceived called.<br><br>For example, here's a slightly stripped pair of protocols (server and<br>client) that I'm currently using as part of a bigger project. Most of
<br>the communication is over a PB connection which the client uses to<br>perform operations on the server, one of which is editing job<br>information. But jobs contain attached files (often very large<br>audio/video files), so committing changes to a job also involves
<br>transmitting up any newly added files. So after the client updates<br>the server's meta data, it initiates a separate set of file transfers<br>across a different port.<br><br>In my case, the header for a file transfer includes a session key
<br>(which the protocol uses to reference the original PB-based job<br>session the client was using) along with a file key used for storage<br>(which uniquely references a specific file in the job). The final<br>element is the total file size. That is, upon connecting, the client
<br>transmits a line such as:<br><br> <session_uuid> <file_uuid> #######<br><br>where the two uuids are specific to the transfer underway (and help<br>with security since a random client isn't going to know the right
<br>ids), and ######## is the overall file length. After sending that<br>line (e.g., right after its final newline), the client just blasts up<br>the raw data.<br><br>The protocol is a simple LineReceiver based protocol, that receives
<br>that information information as an ASCII initial line, after which it<br>switches to raw mode to receive the data. Although the data length<br>could technically be inferred from when the client disconnects, having<br>
it up front ensures I can detect a transfer that gets interrupted.<br><br>So on the server side you have:<br><br> - - - - - - - - - - - - - - - - - - - - - - - - -<br><br>class FileIOProtocol(LineReceiver):<br><br>
def __init__(self):<br> <a href="http://self.info">self.info</a> = None<br> self.outfile = None<br> self.remain = 0<br> self.crc = 0<br><br> def lineReceived(self, line):<br> logger.debug
('FileIOProtocol:lineReceived:%s', line)<br> sess_key, file_key, self.size = line.split()<br> file_key = uuid.UUID(file_key)<br><br> try:<br> session_uuid = uuid.UUID(sess_key)<br> except:
<br> logger.debug('FileIOProtocol:lineReceived Invalid session')<br> self.transport.loseConnection()<br> return<br><br> self.job_session = self.factory.sessions.get(session_uuid)
<br> if not self.job_session:<br> logger.debug('FileIOProtocol:lineReceived Invalid session')<br> self.transport.loseConnection()<br> return<br><br> if not self.job_session.active:
<br> logger.debug('FileIOProtocol:lineReceived Stale session')<br> self.transport.loseConnection()<br> return<br><br> # [db3l] The original code validates the individual file uuid here
<br> # resulting in self.job_file as job file object from the session<br><br> if not self.job_file:<br> logger.debug('FileIOProtocol:lineReceived Invalid file key')<br> self.transport.loseConnection
()<br> return<br><br> # Create the upload directory if not already present<br> if not os.path.isdir(self.job_session.upload_dir):<br> os.makedirs(self.job_session.upload_dir)<br><br>
self.outfilename = os.path.join(self.job_session.upload_dir,<br> self.job_file['uuid'].hex)<br><br> logger.debug('FileIOProtocol:lineReceived Receiving into %s',
<br> self.outfilename)<br> try:<br> self.outfile = open(self.outfilename,'wb')<br> except Exception, value:<br> logger.debug('FileIOProtocol:lineReceived Unable to open file %s '
<br> '(%s)', self.outfilename, value)<br> self.transport.loseConnection()<br> return<br><br> self.remain = int(self.size)<br> logger.debug('FileIOProtocol:lineReceived Entering raw mode: %s %s',
<br> self.outfile, self.remain)<br> self.setRawMode()<br><br> def rawDataReceived(self, data):<br> self.remain -= len(data)<br> self.crc = crc32(data, self.crc)<br> self.outfile.write
(data)<br><br> def connectionMade(self):<br> LineReceiver.connectionMade(self)<br> logger.debug('FileIOProtocol:connectionMade')<br><br> def connectionLost(self, reason):<br> LineReceiver.connectionLost
(self, reason)<br> logger.debug('FileIOProtocol:connectionLost')<br> if self.outfile:<br> self.outfile.close()<br><br> if self.remain != 0:<br> # Problem uploading - discard
<br> logger.debug('FileIOProtocol:connectionLost remain(%d)!=0',<br> self.remain)<br><br> os.remove(self.outfilename)<br> else:<br> # Update job object with upload status
<br> self.job_file['uploaded'] = datetime.utcnow()<br> self.job_file['size'] = self.size<br> self.job_file['crc'] = self.crc<br><br><br>class FileIOFactory(ServerFactory):
<br> protocol = FileIOProtocol<br><br> def __init__(self, db, sessions, options):<br> self.db = db<br> self.options = options<br> self.sessions = sessions<br><br> - - - - - - - - - - - - - - - - - - - - - - - - -
<br><br>which is bound to an appropriate port on the server however you'd like.<br>I use code like:<br><br> self.fileio = FileIOFactory(db, self.sessions, options)<br> reactor.listenTCP(self.options['file_port'],
self.fileio)<br><br><br>On the client side, I have an equivalent protocol that transmits up<br>the file. It's run beneath a GUI, so keeps a reference to the GUI<br>controller object that might indicate it needs to cancel a transfer
<br>mid-stream, as well as updating the controller during the transfer so<br>it can update a progress bar on screen.<br><br>It is also a LineReceiver based protocol, and uses the Twisted<br>FileSender object to do the raw data transfer (which is implemented as
<br>a straight producer with the TCP socket being the consumer). The<br>connectionMade method is where it transmits the ASCII header and then<br>institutes the raw data transfer.<br><br> - - - - - - - - - - - - - - - - - - - - - - - - -
<br><br><br>class TransferCancelled(Exception):<br> """Exception for a user cancelling a transfer"""<br> pass<br><br>class FileIOClient(LineReceiver):<br><br> def __init__(self, path, sess_key, file_key, controller):
<br> self.path = path<br> self.sess_key = sess_key<br> self.file_key = file_key<br> self.controller = controller<br><br> self.infile = open(self.path, 'rb')<br> self.insize
= os.stat(self.path).st_size<br><br> self.result = None<br> self.completed = False<br><br> self.controller.file_sent = 0<br> self.controller.file_size = self.insize<br><br> def _monitor(self, data):
<br> self.controller.file_sent += len(data)<br> self.controller.total_sent += len(data)<br><br> # Check with controller to see if we've been cancelled and abort<br> # if so.<br> if self.controller.cancel
:<br> print 'FileIOClient._monitor Cancelling'<br> # Need to unregister the producer with the transport or it will<br> # wait for it to finish before breaking the connection<br>
self.transport.unregisterProducer()<br> self.transport.loseConnection()<br> # Indicate a user cancelled result<br> self.result = TransferCancelled('User cancelled transfer')<br><br>
return data<br><br> def cbTransferCompleted(self, lastsent):<br> self.completed = True<br> self.transport.loseConnection()<br><br> def connectionMade(self):<br> self.transport.write('%s %s %s\r\n' % (str(
self.sess_key),<br> str(self.file_key),<br> self.insize))<br> sender = FileSender()<br> sender.CHUNK_SIZE = 2 ** 16
<br> d = sender.beginFileTransfer(self.infile, self.transport,<br> self._monitor)<br> d.addCallback(self.cbTransferCompleted)<br><br> def connectionLost(self, reason):<br>
LineReceiver.connectionLost(self, reason)<br> print 'FileIOClient:connectionLost'<br> self.infile.close()<br> if self.completed:<br> self.controller.completed.callback(self.result
)<br> else:<br> self.controller.completed.errback(reason)<br><br>class FileIOClientFactory(ClientFactory):<br><br> protocol = FileIOClient<br><br> def __init__(self, path, sess_key, file_key, controller):
<br> self.path = path<br> self.sess_key = sess_key<br> self.file_key = file_key<br> self.controller = controller<br><br> def clientConnectionFailed(self, connector, reason):<br> ClientFactory.clientConnectionFailed
(self, connector, reason)<br> self.controller.completed.errback(reason)<br><br> def buildProtocol(self, addr):<br> print 'buildProtocol'<br> p = self.protocol(self.path, self.sess_key, self.file_key
,<br> self.controller)<br> p.factory = self<br> return p<br><br> - - - - - - - - - - - - - - - - - - - - - - - - -<br><br><br>Within the presentation layer controller on the client, initiating a
<br>transfer is done with:<br><br> def _transmitOne(self, address, port, path, sess_key, file_key):<br> self.completed = defer.Deferred()<br> f = FileIOClientFactory(path, sess_key, file_key, self)<br>
reactor.connectTCP(address, port, f)<br> return self.completed<br><br>and the result is that self.completed fires (callback or errback) when<br>the transfer is done (which the controller uses to then initiate the<br>
next transfer when there are a list of files to go up for a job).<br><br>While probably not exactly what you're trying to do, perhaps it'll<br>point you in the right direction.<br><br>-- David<br><br><br>_______________________________________________
<br>Twisted-Python mailing list<br><a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br><a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
</a><br></blockquote></div><br>