[Twisted-Python] Making twisted web client requests on running reactor

Ameya Lokare lokare.ameya at gmail.com
Sun Feb 23 17:06:09 MST 2014


Thanks, Hynek and Jean-Paul! Crochet looks like exactly what I'm looking
for.

-Ameya


On Sun, Feb 23, 2014 at 5:11 AM, <exarkun at twistedmatrix.com> wrote:

> On 02:17 am, lokare.ameya at gmail.com wrote:
>
>> Hi everyone,
>>
>> I'm writing a client library that makes (potentially long running) HTTP
>> requests. Since the library will be used within non-Twisted code, I was
>> thinking of running the reactor in a separate thread. The library code
>> looks roughly like this:
>>
>> def start():
>>    Thread(target=reactor.run, args=(False,)).start()
>>
>> def shutdown():
>>    reactor.callFromThread(lambda x: reactor.stop(), None)
>>
>> def make_request(url):
>>        agent = Agent(reactor)
>>        d = agent.request(
>>            'GET',
>>            url,
>>            Headers({
>>                'User-Agent': ['Twisted SSE Client'],
>>                'Cache-Control': ['no-cache'],
>>            }),
>>            None)
>>        d.addCallback(self.cbRequest)
>>
>> Then I make requests from the application like so:
>>
>> start()
>> make_request("http://localhost:12000")
>> ....
>> make_request("http://localhost:12000")
>> ...
>> shutdown()
>>
>> However, it looks like the 'd' deferred never gets fired. From a little
>> playing around, it looks like this is because the reactor is started
>> before
>> the call(s) to agent.request are made. All examples in the twisted docs
>> create the request before running the reactor. I'm sure I'm missing
>> something very obvious here. Also, is there a better way to design such a
>> library?
>>
>
> It sounds like you want to use Crochet (which Hynek already linked you to).
>
> The reason your code doesn't work, though, is that Twisted APIs are not
> thread safe.  You may *only* call them in the thread the reactor is running
> in.
>
> Your `start` function runs the reactor in a new thread.  Then the code
> proceeds to try to use `Agent` from the original thread.  This is
> unsupported.
>
> Jean-Paul
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
-Ameya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140223/0b60a6f7/attachment.html>


More information about the Twisted-Python mailing list