[Twisted-Python] twisted thumbnail server

[Paul Wiseman]

On 2 November 2012 15:57, Laurens Van Houtven <_ at lvh.cc> wrote:

> Yeah, very big +1 to showing the deferToThread version. I feel bad even
> trying to spot potential threading issues here... It could be because the
> default thread pool isn't very large, but you're making many requests.
>
> What functionality does boto have that txaws doesn't that you really need
> here? Perhaps you can avoid blocking (and hence threads) at all.
>
>
This is the deferToThread version:
https://github.com/GP89/thumbs/blob/defertothread/thumb.py

The thread pool size was 60, pretty arbitrary but (ignoring the memory
leaks) it should only have at most n images in memory at once where n is
the number of threads. If there's more requests that it can process they'll
just build up in the queue and not cause memory to fill up with downloaded
images (this was an initial problem I had).

My understanding is that txaws would only work in a twisted way, but as the
work in the thread has blocking code (the PIL bits) I just used boto as I'm
more familiar with it. I'm not sure how I'd use txaws in a thread or what
benefit that would have?

I'd love to avoid blocking and keep it all in 1 thread, but I don't know of
anyway to do the image resizing/rotation etc. without blocking.


>
> On Fri, Nov 2, 2012 at 4:42 PM, Paul Wiseman <poalman at gmail.com> wrote:
>
>> I hope this will be an easy question for some of you guys :)
>>
>> I'm trying to set up a simple server which will accept requests over GET
>> to create a thumbnail for an image, and server it back as the response.
>>
>> The images are stored in two S3 buckets, the originals are in one bucket
>> (store), and the generated thumbnails are stored in another (thumb) as a
>> cache so that the work doesn't need to be repeated.
>>
>> Currently I'm checking if the thumbnail already exists in the thumb
>> bucket. I'm redirecting the request if it is or if not I'm downloading the
>> image from store, generating the thumb using PIL, uploading the thumbnail
>> to the thumb bucket and then redirecting the request.
>>
>> I'm very new to twisted and was wondering if anyone who is more
>> experienced would be able to take a look at what I have so far and let me
>> know if anything is wrong/not ideal/will cause problems etc. or just
>> general style pointers? The more critical the better, as I said I'm very
>> new to this.
>>
>> I've just chucked it up on github:
>> https://github.com/GP89/thumbs/blob/master/thumb.py
>>
>> There's a definite memory leak right now which I believe is PIL, or
>> possibly StringIO objects not being disposed, hence all the random del
>> statements trying to cure it (unsuccessfully). Maybe there's something I'm
>> doing wrong in twisted that is causing things not to be cleaned up that I'm
>> not aware of as well.
>>
>> I did try to use deferToThread, rather than my thread pool but the server
>> seemed to block up- I probably should have left it incase it was because I
>> was doing something obviously wrong. I think I'll make a branch quickly
>> with my deferToThread version.
>>
>> Thanks very much for any time you can lend!
>>
>> Paul
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
>
> --
> cheers
> lvh
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20121102/245957f1/attachment.htm