[Twisted-Python] Asynchronously reading posted data

Paul Wiseman poalman at gmail.com
Thu Sep 5 02:50:26 MDT 2013


Hey,

Thanks for your reply! The json data should never be too long so I'm not
worried about the memory usage, I need the whole json object to start
working anyway realistically - I was more concerned about blocking reading
the data from the network - specifically the request.content.read(), if the
client happens to be sending it very slowly this would block everything up
right? - or would this not be an issue?

Maybe because I have fairly small content bodies I wont have to worry?

Payl


On 4 September 2013 22:45, Glyph <glyph at twistedmatrix.com> wrote:

>
> On Sep 4, 2013, at 10:48 AM, Paul Wiseman <poalman at gmail.com> wrote:
>
> > I'm still pretty new to twisted and feel I'm slowly getting the hang of
> it, enough to realise that this line of code is probably going to block and
> not do me any favours, the line is the first line in a resource.Resource
> render_POST.
> >
> > json_request = json.loads(request.content.read())
> >
> > The resource.Resource is a child of another resource which is passed to
> server.Site which is passed to internent.TCPServer.
> >
> > The problem is I can't work out how I can read the post data from the
> request in an async way.
> >
> > I have a feeling I need to implement a protocol, I'm guessing a
> LineReceiver but I can't figure out how I'd tie that in with my current
> solution or specifically how the LineReceiver would even read
> asynchronously to be honest..
> >
> > Maybe the read is fine? I need the whole post data to do anything useful
> I guess as I can't string to a json decoder that I'm aware of. Just it will
> block everything up while I read, which shouldn't be long but I guess I'm
> bound to the speed of the person posting.
> >
> > Thanks all!
>
> If you're parsing a JSON object, you're going to be representing the whole
> thing in memory at the end of the interaction regardless, even if you
> parsed it and buffered it in an event-driven way.
>
> That means you need to keep this data relatively small no matter what; if
> it's arbitrarily large, you are going to start swapping anyway.
>
> So probably, just doing the blocking parse is fine.  You might be able to
> save a *little* memory by parsing it as it comes in, but you're also going
> to have to write your own JSON parser, which is probably going to take more
> programmer time than you will ever save in execution time by this marginal
> memory reduction :-).  Better would be to spend that effort enforcing
> stringent resource limits so that you will give up reading before you ever
> get to the parse in the case where it's big enough to cause a problem.
>
> -glyph
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130905/924eaa60/attachment-0001.html>


More information about the Twisted-Python mailing list