[Twisted-Python] High throughput database logger

[Adi Libotean]

Hi,

I'm looking at various options for implementing a high throughput 
database logger that will work with Twisted.

My requirements, listed by importance:

1) small memory footprint
2) high speed
3) low garbage generation

The application I'm working on runs continuously (24/7). I've 
experimented a bit with pysqlite and Twisted to see which approach is 
better suited (see attached example).

----

Question 1: I noticed that all of the Twisted based versions are very 
slow compared to the plain sqlite3 test. This seems to be caused by 
atomic transaction management, namely a commit after each insert.

Would be interested to know if there is a simple way to avoid this and 
do my own transaction management (aka batch commit).

One other thing is the greatly varying amounts of garbage generated 
(peak memory) and memory usage between the Twisted variants.

----

Question 2: I would have expected B (Twisted ADBAPI) to behave very 
similar to C/E since I'm using a connection pool of size 1 and all 
requests are queued and handled sequentially.

Could any of you please give me some pointers as to why this is happening?

----

Question 3: Even though objgraph lists the exact same reference count 
once the code is ran, the amount of used memory greatly differs. Any 
ideas what might be causing this?

Any suggestions and/or pointers on how to improve/do this are more than 
welcome.

Thank you for your time,
Adrian
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: db-insert-test.py
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140904/1d611721/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: results.txt
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140904/1d611721/attachment.txt>