[Twisted-Python] Asynchronously reading posted data

[Glyph]

On Sep 4, 2013, at 10:48 AM, Paul Wiseman <poalman at gmail.com> wrote:

> I'm still pretty new to twisted and feel I'm slowly getting the hang of it, enough to realise that this line of code is probably going to block and not do me any favours, the line is the first line in a resource.Resource render_POST.
> 
> json_request = json.loads(request.content.read())
> 
> The resource.Resource is a child of another resource which is passed to server.Site which is passed to internent.TCPServer.
> 
> The problem is I can't work out how I can read the post data from the request in an async way.
> 
> I have a feeling I need to implement a protocol, I'm guessing a LineReceiver but I can't figure out how I'd tie that in with my current solution or specifically how the LineReceiver would even read asynchronously to be honest..
> 
> Maybe the read is fine? I need the whole post data to do anything useful I guess as I can't string to a json decoder that I'm aware of. Just it will block everything up while I read, which shouldn't be long but I guess I'm bound to the speed of the person posting.
> 
> Thanks all!

If you're parsing a JSON object, you're going to be representing the whole thing in memory at the end of the interaction regardless, even if you parsed it and buffered it in an event-driven way.

That means you need to keep this data relatively small no matter what; if it's arbitrarily large, you are going to start swapping anyway.

So probably, just doing the blocking parse is fine.  You might be able to save a *little* memory by parsing it as it comes in, but you're also going to have to write your own JSON parser, which is probably going to take more programmer time than you will ever save in execution time by this marginal memory reduction :-).  Better would be to spend that effort enforcing stringent resource limits so that you will give up reading before you ever get to the parse in the case where it's big enough to cause a problem.

-glyph