On 2 November 2012 16:13, Phil Mayers <span dir="ltr"><<a href="mailto:p.mayers@imperial.ac.uk" target="_blank">p.mayers@imperial.ac.uk</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 02/11/12 15:42, Paul Wiseman wrote:<br>
> I hope this will be an easy question for some of you guys :)<br>
><br>
> I'm trying to set up a simple server which will accept requests over GET<br>
> to create a thumbnail for an image, and server it back as the response.<br>
><br>
> The images are stored in two S3 buckets, the originals are in one bucket<br>
> (store), and the generated thumbnails are stored in another (thumb) as a<br>
> cache so that the work doesn't need to be repeated.<br>
><br>
> Currently I'm checking if the thumbnail already exists in the thumb<br>
> bucket. I'm redirecting the request if it is or if not I'm downloading<br>
> the image from store, generating the thumb using PIL, uploading the<br>
> thumbnail to the thumb bucket and then redirecting the request.<br>
<br>
</div>This isn't a criticism, but I trust you are aware of the implications<br>
and problems of doing work in threads?</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> </blockquote><div><br></div><div>I think so. I understand the whole idea of twisted is to schedule tasks in an async way in a single main thread. I usually use quite a lot of threads in my code and I'm just learning about this new async style of coding. I've tried to avoid using threads as much as I can here but didn't think I could get away from them based on the fact that PIL is blocking.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
FWIW we usually use a child process pool for intensive tasks; this has<br>
the advantage you can sensibly kill a long-lived child (just kill the<br>
process) and you side-step the lack of concurrency in the python<br>
interpreter.<br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
[In this case, I'd just start up a bunch of python interpreters using a<br>
ProcessProtocol and use a simple request/response command protocol on<br>
stdin/stdout - the child interpreters can be non-Twisted processes able<br>
to block on PIL operations]<br>
<br></blockquote><div><br></div><div><div>I'd love to get this working using a processes pool rather than a thread pool (I spent quite a lot of time trying to figure out how to do it in twisted but haven't yet worked out how). As this server will be CPU bound this will hopefully get more throughput and also side-step the memory leak in PIL that I believe I'm seeing (although on second thought maybe not, the over head of starting a new python interpreter each time wouldn't be viable).</div>
<div><br></div><div>Are there any examples of how to use a process pool?</div></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If you really do want threads, is there any reason to not use the<br>
Twisted threadpool stuff?<br>
<br></blockquote><div><br></div><div>I wasn't aware of a twisted thread pool, I think I've only come across deferToThead, which I imagined was using a threadpool. (if so if there a way to control the size?)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It's often a personal/style choice, but I don't use StringIO for large<br>
volumes of data personally (not Twisted-specific).<br>
<br></blockquote><div><br></div><div>I rarely use it either, but I needed a way to get the data into a file type object for PIL without putting the data to disk. How would you recommend I get the data between download / PIL / upload?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm sure someone will mention tests ;o) </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"> </div></div></blockquote>
<div><br></div><div>tests are something I've often, guiltily, neglected- maybe I should start :) I wouldn't really know where to start writing tests for anything more than trivial functions. I guess just have images and thumbnails that I know are correct for that request, send them to the server and see if they return a thumbnail that matches? something along those lines? </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</div></div></blockquote></div><br>