[Twisted-Python] SQLAlchemy and Twisted

[Daniel Griffin]

That is kind of what I found. In my app I basically have the following
triggered using callLater:

do a select and create a new SSL connection  for each item returned
Wait for the response
Do between 2 and 4 inserts depending on results
Close the connection.

As measured in connections completed per seconds my results have been:

15/second with blocking twisted
7/second using deferred to thread
18/second using SQLalchemy in another process over PB.

I know these are anecdotal but they might help someone else.

The two problems with PB are that you quickly overrun the maximum amount of
FD's available for select reactors and you end up roughly doubling the
amount of DB querying you are doing.

I intend to try using something like ampoule(deferToProcess) but I think I
will have the same problems as with PB since it just uses TCP sockets
anyways.


Dan



On Sun, May 9, 2010 at 4:45 AM, Phil Mayers <p.mayers at imperial.ac.uk> wrote:

> On 05/09/2010 07:07 AM, Daniel Griffin wrote:
> > If you let SQLAlchemy block twisted would there be any impact besides
> > performance?
>
> Depends how long it blocks for, and what else your process is doing.
>
> With the reactor blocked:
>
>  * no socket reads or accepts can be done
>  * no callLater or LoopingCall can be scheduled
>
> ...and so on.
>
> If you've only got a small number of clients and no time-based
> scheduling and you're only blocking for a short time (tens or hundreds
> of milliseconds, say) it may be tolerable.
>
> If the server has a lot of clients, or needs to accept new ones in a
> timely fashion, or has time-based work to do, and your blocking might go
> on for a while, it's probably not tolerable.
>
> I'd avoid it personally (but then I'm utterly unconvinced by ORMs) and
> stick to using SA in a thread as your SQL query layer.
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100509/9c828ec3/attachment-0001.html>