[Twisted-Python] twisted thumbnail server

Phil Mayers p.mayers at imperial.ac.uk
Fri Nov 2 12:13:37 EDT 2012


On 02/11/12 15:42, Paul Wiseman wrote:
> I hope this will be an easy question for some of you guys :)
>
> I'm trying to set up a simple server which will accept requests over GET
> to create a thumbnail for an image, and server it back as the response.
>
> The images are stored in two S3 buckets, the originals are in one bucket
> (store), and the generated thumbnails are stored in another (thumb) as a
> cache so that the work doesn't need to be repeated.
>
> Currently I'm checking if the thumbnail already exists in the thumb
> bucket. I'm redirecting the request if it is or if not I'm downloading
> the image from store, generating the thumb using PIL, uploading the
> thumbnail to the thumb bucket and then redirecting the request.

This isn't a criticism, but I trust you are aware of the implications 
and problems of doing work in threads?

FWIW we usually use a child process pool for intensive tasks; this has 
the advantage you can sensibly kill a long-lived child (just kill the 
process) and you side-step the lack of concurrency in the python 
interpreter.

[In this case, I'd just start up a bunch of python interpreters using a 
ProcessProtocol and use a simple request/response command protocol on 
stdin/stdout - the child interpreters can be non-Twisted processes able 
to block on PIL operations]

If you really do want threads, is there any reason to not use the 
Twisted threadpool stuff?

It's often a personal/style choice, but I don't use StringIO for large 
volumes of data personally (not Twisted-specific).

I'm sure someone will mention tests ;o)



More information about the Twisted-Python mailing list