On 2 November 2012 15:57, Laurens Van Houtven <span dir="ltr"><<a href="mailto:_@lvh.cc" target="_blank">_@lvh.cc</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yeah, very big +1 to showing the deferToThread version. I feel bad even trying to spot potential threading issues here... It could be because the default thread pool isn't very large, but you're making many requests.<br>
<br>What functionality does boto have that txaws doesn't that you really need here? Perhaps you can avoid blocking (and hence threads) at all.<br><div class="gmail_extra"><br></div></blockquote><div><br></div><div>This is the deferToThread version: <a href="https://github.com/GP89/thumbs/blob/defertothread/thumb.py">https://github.com/GP89/thumbs/blob/defertothread/thumb.py</a></div>
<div><br></div><div>The thread pool size was 60, pretty arbitrary but (ignoring the memory leaks) it should only have at most n images in memory at once where n is the number of threads. If there's more requests that it can process they'll just build up in the queue and not cause memory to fill up with downloaded images (this was an initial problem I had).</div>
<div><br></div><div>My understanding is that txaws would only work in a twisted way, but as the work in the thread has blocking code (the PIL bits) I just used boto as I'm more familiar with it. I'm not sure how I'd use txaws in a thread or what benefit that would have?</div>
<div><br></div><div>I'd love to avoid blocking and keep it all in 1 thread, but I don't know of anyway to do the image resizing/rotation etc. without blocking.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Fri, Nov 2, 2012 at 4:42 PM, Paul Wiseman <span dir="ltr"><<a href="mailto:poalman@gmail.com" target="_blank">poalman@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">I hope this will be an easy question for some of you guys :)<div><br></div><div>I'm trying to set up a simple server which will accept requests over GET to create a thumbnail for an image, and server it back as the response.</div>
<div><br></div><div>The images are stored in two S3 buckets, the originals are in one bucket (store), and the generated thumbnails are stored in another (thumb) as a cache so that the work doesn't need to be repeated.</div>
<div><br></div><div>Currently I'm checking if the thumbnail already exists in the thumb bucket. I'm redirecting the request if it is or if not I'm downloading the image from store, generating the thumb using PIL, uploading the thumbnail to the thumb bucket and then redirecting the request.</div>
<div><br></div><div>I'm very new to twisted and was wondering if anyone who is more experienced would be able to take a look at what I have so far and let me know if anything is wrong/not ideal/will cause problems etc. or just general style pointers? The more critical the better, as I said I'm very new to this.</div>
<div><br></div><div>I've just chucked it up on github: <a href="https://github.com/GP89/thumbs/blob/master/thumb.py" target="_blank">https://github.com/GP89/thumbs/blob/master/thumb.py</a> </div><div><br></div><div>There's a definite memory leak right now which I believe is PIL, or possibly StringIO objects not being disposed, hence all the random del statements trying to cure it (unsuccessfully). Maybe there's something I'm doing wrong in twisted that is causing things not to be cleaned up that I'm not aware of as well.</div>
<div><br></div><div>I did try to use deferToThread, rather than my thread pool but the server seemed to block up- I probably should have left it incase it was because I was doing something obviously wrong. I think I'll make a branch quickly with my deferToThread version.</div>
<div><br></div><div>Thanks very much for any time you can lend!</div><span><font color="#888888"><div><br></div><div>Paul</div>
</font></span><br></div></div>_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com" target="_blank">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br>cheers<div>lvh</div><br>
</font></span></div>
<br>_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br></blockquote></div><br>