[Twisted-Python] Merging databases the Twisted way

[Gerhard Haering]

On Wed, Oct 13, 2004 at 02:36:18PM +0200, Thomas Weholt wrote:
> Hi,
> 
> I got a server running a SQLite-database. Sometimes I need to merge
> data from another SQLite database into the master. So far I've done
> blocking old-style read-some from source, write-some to destination
> stuff using the standard pysqlite-database module, but as mentioned,
> it blocks the server.

If you haven't done so already, maybe you can try to ATTACH (SQLITE
SQL command) the other database and then merge using SQL commands.
This should be faster and easier than going through Python.

> AFAIK SQLite only allows one connection to the database as well so
> that might make it harder to implement a non-blocking solution.
> [...]

Basically, one open transaction per database. Concurrent access will
be blocked, for as long as the timeout parameter of the connect()
call.

With PySQLite2/SQLite3 it will be much better, because SQLite3 has
much more fine-grained locking.

SQLite also features an API call to register a callback that will be
invoked every n SQLite VM operations. I'll wrap it one day, maybe the
Twisted folks can make some use of it for a new SQLiteReactor or
whatever.

-- Gerhard (subscribed, but still hasn't used Twisted for real, yet)