[Twisted-Python] Twisted as a full solution for web hosting [WSGI + other]

[exarkun at twistedmatrix.com]

On 07:41 am, orestis at orestis.gr wrote:
>On 21 Οκτ 2013, at 10:32 μ.μ., Glyph <glyph at twistedmatrix.com> wrote:
>>
>>On Oct 20, 2013, at 2:21 AM, Orestis Markou <orestis at orestis.gr> 
>>wrote:
>>>Hello,
>>>
>>>Short form of the question:
>>>
>>>Are people using Twisted to host WSGI applications AND serve static 
>>>files AND replace celery/redis/other?
>>
>>I'm not personally using it as a WSGI host, but otherwise, yes, a 
>>full-stack application container speaking multiple protocols.
>
>Any pointers on how to best use this in combination with WSGI/Django? 
>In the past I had a combination of twisted-web (for /static and /media) 
>and wsgi host (for everything else), all running under the same 
>Service. Essentially:
>
>os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
>application = django.core.handlers.wsgi.WSGIHandler()
>
>wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), 
>application)
>resource = Root(wsgi_resource)
># this could probably be automatically inferred from settings.py
>resource.putChild('static', File(...))
>resource.putChild('uploaded', File(...))
># other stuff
>site = server.Site(resource)
>reactor.listenTCP(8000, site)
>reactor.run()

This looks about like I'd expect (if my guess that `Root` is an 
`IResource` that knows how to split dispatch between the WSGI resource 
and its other children).  If you have any suggestions for improving the 
experience please share them. :)
>[snip]
>
>Here's a thought experiment - I'd like to keep URL routing 100% in 
>Django for anything that hits the DB. I have some code that needs to 
>spawn an external process to generate an image on-demand (with a layer 
>of caching on top). In the past I defined a Twisted.Web handler for 
>that. Could I now expose an internal API that (through Crochet?) do the 
>spawnProcess dance and come back with image data that Django could then 
>handle internally (store in a file, whatever). How would the threaded 
>WSGI container deal with the blocking request (not really blocking, but 
>that request would stall until the Deferred would be fired).

It will produce roughly the same results as you'd get if you used any 
other WSGI container: one of the threads in the thread pool will be kept 
unavailable as it waits for the result.  This will have the usual 
consequence: if your threadpool has a max of N threads and you receive N 
requests that need to do this, the N+1th request that needs to be 
handled by the WSGI part of your server won't be handled until one of 
the previous requests completes (completion frees up one thread which is 
then used to handle the N+1th request).

The only difference might be that since you also have non-WSGI content 
(all of your static content) even when your thread pool is completely in 
use requests for static content will still be satisfied right away. 
However, if you previously had a WSGI+lighttpd/whatever then you 
probably already had this property as well.

Put another way, Twisted's WSGI container is still just a WSGI 
container.  Fortunately Twisted has other pieces aside from a WSGI 
container. :)

Jean-Paul