[Twisted-Python] Re: writing back to the connection is blocking my code

[Thomas Hervé]

Quoting coder_gus <coder_gus at lavabit.com>:

> Hi, thanks for answering and clarifying this - until now you where the
> only one that told me there no way of sending data if there is no event
> on the server. So what I am trying to achieve, I have asked a couple of
> weeks ago too about this matter (actually about the architecure): I
> want to write an application server that receives data from clients
> (client app written in delphi), data is in xml format (I want to write
> it using a bare-bone protocol and not an xml-rpc server). The data the
> client sends are some requests for some specific data hold in a
> database. So, the client makes a request, request that is hold in a
> synchronized queue together with an instance of the client's
> connection. A coordinator takes the data from that queue and gives it
> to some xml threads to parse it and the same coordinator take the
> parsed data from the xml threads and give some workers to do the
> bussiness work. This workers place the result in a queue from where the
> coordinator takes care of sending it back to the clients. I will be
> using sqlalchemy for database communication.
> I know the architecture is pretty weird from the twisted perspective,
> but when the dev-manager wants to do it that way, what can you do? So
> that is the thing I want to achieve and from this started my question;
> when the workers finish their work, they have the result, place it in a
> queue and from there I have to send it to the clients asap, without
> waiting the server for events.

This is were you're wrong: pushing the result is an event! So to the  
event 'I got some result in a thread', you'll fire a connection write.  
Let me try to write some pseudo code to solve your problem:


def blockingTask(content):
     import sqlalchemy
     # sqlalchemy stuff
     query = session.query("bla")
     ....
     return result


class CommandProtocol(Protocol):
     def __init__(self):
         self.buf = ''

     def dataReceived(self, data):
         self.buf += data
         # This is stupid, don't do that
         while '</end_command>' in self.buf:
              content, self.buf = self.buf.split('</end_command>', 1)
              content += '</end_command>'
              deferToThread(blockingTask, content).addCallback(self.response)

     def response(self, result):
         # I got the response event!
         self.sendLine.write("response %s" % (result,))


See? deferToThread generates an event, which is used the pass the  
result of the command to a callback.


I hope it clarifies some things.

-- 
Thomas