[Twisted-Python] question about threading

glyph at divmod.com glyph at divmod.com
Tue Oct 25 11:27:51 EDT 2005

On Tue, 25 Oct 2005 11:05:21 -0400, Phil Christensen <phil at bubblehouse.org> wrote:
>hey folks,

Let me start from the other end of your message:

>the client object holds a reference to the protocol object, and  sendCommand 
>basically just executes:
>         print "Sending " + message + " to " + str(self.protocol.source)
>         self.protocol.transport.write(message + "\0")
The reactor is not thread safe, and thus this call to 'write' has undefined (and bad) behavior.  You _MUST_ use the reactor thread APIs; they're not optional conveniences, they're the only way you can use threads with Twisted :)

>i need to spawn a thread 

danger, danger will robinson.  Here is where your trouble started :).  Actually you *don't* need to spawn a thread, you need to spawn a process - twisted supports processes - and I can't guarantee that os.system will work properly from within a Twisted application.  Handling of SIGCHILD has proven to be a sticky wicket in the past.

>i know there's a bunch of reactor methods that deal with threads, but  i've 
>never used them before, and i'm not sure which one will fix this  issue."

There are really only 2: callInThread and callFromThread.  There is a convenience API, twisted.internet.threads.deferToThread, which might be what you wanted, if what you wanted was in fact a thread.  However, you want spawnProcess in any event.  As I said, os.system may not work at _all_ from within Twisted, depending on your operating system.

>here's my thread subclass:

Don't subclass thread.  Twisted implements its own threadpool; use callInThread with what used to be your 'run' function.

>     def run(self):
>         pres = event.getActivePresentation()
>         t = datetime.datetime.now()
>         base_name = 'sample-file-name'
>         out_html = file('/tmp/' + base_name + '.html', 'w')
>         # [snip snip snip]
>         # write some html to the file
>         out_html.close()

          This bit could actually be threaded, if it's slow and blocking.  My suggestion: If it's for a demo, just block.

>         os.system('html2ps /tmp/' + base_name + '.html > /tmp/' +  base_name 
>+ '.ps')  # > /tmp/' + base_name + '_ps.log')

Since reactor.spawnProcess might be a bit tedious for simply running this here, try this:

          twisted.internet.utils.getProcessOutput('/usr/bin/html2ps', ['html2ps', ...).addCallback(keepGoing)

>         os.system('mkdir data/transcripts/' + str(pres.id))
          ^ UGH!  Why are you spawning another process here??  os.mkdir, please.
>         os.system('ps2pdf /tmp/' + base_name + '.ps data/ transcripts/' + 
>str(pres.id) + '/' + base_name + '.pdf > /tmp/' +  base_name + '_pdf.log')
          ^ Another getProcessOutput here.  Return the resultant Deferred from within your keepGoing callback...

>         lock = threading.Lock()
>         lock.acquire()
          ^ Hooray, now you can forget about this garbage

>         self.client.sendCommand('presentTranscript', ['/ transcripts/' + 
>str(pres.id) + '/' + base_name + '.pdf'])
>         lock.release()
          ^ Do this in the final callback of the Deferred that you've just created.

I hope this helped.

More information about the Twisted-Python mailing list