[Twisted-Python] twisted thumbnail server

Laurens Van Houtven _ at lvh.cc
Fri Nov 2 11:57:42 EDT 2012


Yeah, very big +1 to showing the deferToThread version. I feel bad even
trying to spot potential threading issues here... It could be because the
default thread pool isn't very large, but you're making many requests.

What functionality does boto have that txaws doesn't that you really need
here? Perhaps you can avoid blocking (and hence threads) at all.


On Fri, Nov 2, 2012 at 4:42 PM, Paul Wiseman <poalman at gmail.com> wrote:

> I hope this will be an easy question for some of you guys :)
>
> I'm trying to set up a simple server which will accept requests over GET
> to create a thumbnail for an image, and server it back as the response.
>
> The images are stored in two S3 buckets, the originals are in one bucket
> (store), and the generated thumbnails are stored in another (thumb) as a
> cache so that the work doesn't need to be repeated.
>
> Currently I'm checking if the thumbnail already exists in the thumb
> bucket. I'm redirecting the request if it is or if not I'm downloading the
> image from store, generating the thumb using PIL, uploading the thumbnail
> to the thumb bucket and then redirecting the request.
>
> I'm very new to twisted and was wondering if anyone who is more
> experienced would be able to take a look at what I have so far and let me
> know if anything is wrong/not ideal/will cause problems etc. or just
> general style pointers? The more critical the better, as I said I'm very
> new to this.
>
> I've just chucked it up on github:
> https://github.com/GP89/thumbs/blob/master/thumb.py
>
> There's a definite memory leak right now which I believe is PIL, or
> possibly StringIO objects not being disposed, hence all the random del
> statements trying to cure it (unsuccessfully). Maybe there's something I'm
> doing wrong in twisted that is causing things not to be cleaned up that I'm
> not aware of as well.
>
> I did try to use deferToThread, rather than my thread pool but the server
> seemed to block up- I probably should have left it incase it was because I
> was doing something obviously wrong. I think I'll make a branch quickly
> with my deferToThread version.
>
> Thanks very much for any time you can lend!
>
> Paul
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>


-- 
cheers
lvh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20121102/d9cb74f8/attachment.htm 


More information about the Twisted-Python mailing list