[Twisted-Python] Making twisted web client requests on running reactor

[exarkun at twistedmatrix.com]

On 02:17 am, lokare.ameya at gmail.com wrote:
>Hi everyone,
>
>I'm writing a client library that makes (potentially long running) HTTP
>requests. Since the library will be used within non-Twisted code, I was
>thinking of running the reactor in a separate thread. The library code
>looks roughly like this:
>
>def start():
>    Thread(target=reactor.run, args=(False,)).start()
>
>def shutdown():
>    reactor.callFromThread(lambda x: reactor.stop(), None)
>
>def make_request(url):
>        agent = Agent(reactor)
>        d = agent.request(
>            'GET',
>            url,
>            Headers({
>                'User-Agent': ['Twisted SSE Client'],
>                'Cache-Control': ['no-cache'],
>            }),
>            None)
>        d.addCallback(self.cbRequest)
>
>Then I make requests from the application like so:
>
>start()
>make_request("http://localhost:12000")
>....
>make_request("http://localhost:12000")
>...
>shutdown()
>
>However, it looks like the 'd' deferred never gets fired. From a little
>playing around, it looks like this is because the reactor is started 
>before
>the call(s) to agent.request are made. All examples in the twisted docs
>create the request before running the reactor. I'm sure I'm missing
>something very obvious here. Also, is there a better way to design such 
>a
>library?

It sounds like you want to use Crochet (which Hynek already linked you 
to).

The reason your code doesn't work, though, is that Twisted APIs are not 
thread safe.  You may *only* call them in the thread the reactor is 
running in.

Your `start` function runs the reactor in a new thread.  Then the code 
proceeds to try to use `Agent` from the original thread.  This is 
unsupported.

Jean-Paul