[Twisted-Python] Twisted as a full solution for web hosting [WSGI + other]

[Orestis Markou]

On 22 Οκτ 2013, at 3:38 μ.μ., exarkun at twistedmatrix.com wrote:

> On 07:41 am, orestis at orestis.gr wrote:
>> On 21 Οκτ 2013, at 10:32 μ.μ., Glyph <glyph at twistedmatrix.com> wrote:
>>> 
>>> On Oct 20, 2013, at 2:21 AM, Orestis Markou <orestis at orestis.gr> wrote:
>>>> Hello,
>>>> 
>>>> Short form of the question:
>>>> 
>>>> Are people using Twisted to host WSGI applications AND serve static files AND replace celery/redis/other?
>>> 
>>> I'm not personally using it as a WSGI host, but otherwise, yes, a full-stack application container speaking multiple protocols.
>> 
>> Any pointers on how to best use this in combination with WSGI/Django? In the past I had a combination of twisted-web (for /static and /media) and wsgi host (for everything else), all running under the same Service. Essentially:
>> 
>> os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
>> application = django.core.handlers.wsgi.WSGIHandler()
>> 
>> wsgi_resource = WSGIResource(reactor, reactor.getThreadPool(), application)
>> resource = Root(wsgi_resource)
>> # this could probably be automatically inferred from settings.py
>> resource.putChild('static', File(...))
>> resource.putChild('uploaded', File(...))
>> # other stuff
>> site = server.Site(resource)
>> reactor.listenTCP(8000, site)
>> reactor.run()
> 
> This looks about like I'd expect (if my guess that `Root` is an `IResource` that knows how to split dispatch between the WSGI resource and its other children).  If you have any suggestions for improving the experience please share them. :)

For anyone following along:

class Root(resource.Resource):

    def __init__(self, wsgi_resource):
        resource.Resource.__init__(self)
        self.wsgi_resource = wsgi_resource

    def getChild(self, path, request):
        path0 = request.prepath.pop(0)
        request.postpath.insert(0, path0)
        return self.wsgi_resource

I don't remember now if I wrote this code or if I copied it from somewhere.

>> [snip]
>> 
>> Here's a thought experiment - I'd like to keep URL routing 100% in Django for anything that hits the DB. I have some code that needs to spawn an external process to generate an image on-demand (with a layer of caching on top). In the past I defined a Twisted.Web handler for that. Could I now expose an internal API that (through Crochet?) do the spawnProcess dance and come back with image data that Django could then handle internally (store in a file, whatever). How would the threaded WSGI container deal with the blocking request (not really blocking, but that request would stall until the Deferred would be fired).
> 
> It will produce roughly the same results as you'd get if you used any other WSGI container: one of the threads in the thread pool will be kept unavailable as it waits for the result.  This will have the usual consequence: if your threadpool has a max of N threads and you receive N requests that need to do this, the N+1th request that needs to be handled by the WSGI part of your server won't be handled until one of the previous requests completes (completion frees up one thread which is then used to handle the N+1th request).
> 
> The only difference might be that since you also have non-WSGI content (all of your static content) even when your thread pool is completely in use requests for static content will still be satisfied right away. However, if you previously had a WSGI+lighttpd/whatever then you probably already had this property as well.
> 
> Put another way, Twisted's WSGI container is still just a WSGI container.  Fortunately Twisted has other pieces aside from a WSGI container. :)

I wonder if this could be turned on its head - instead of having the Django WSGI app blocking while waiting for a long running thing, field the request beforehand, do whatever long-running thing is required and only then hand over the to the WSGI app to proceed with the HTTP response. This obviously requires a level request processing in Twisted - I'm not sure if it's worth the effort. It could be done by a Twisted-savvy WSGI middleware, perhaps. I'll try to see what I can come up with.

Orestis



> Jean-Paul
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20131022/fe78349b/attachment.html>