[Twisted-Python] Considering Twisted for OfflineIMAP

[John Goerzen]

Hello,

I am the author of OfflineIMAP, a multi-threaded multi-account
bi-directional synchronization tool
(http://quux.org/devel/offlineimap) written in Python.

I am running up against several limitations of the current way I'm
doing things.  They include:

1. Lack of thread groups in Python, meaning that if there was a
   network error synchronizing account A, I'd also have to kill off
   the entire application and the synchronizing threads for account B
   -- I couldn't just kill of account A.

2. Various bugs in Python's threading implementation

3. imaplib.py being a stinking festering heap of crap leading to
   unmaintainable code in everything that touches it

My first plan was to just rewrite imaplib, which wouldn't be too
hard.  But someone on the OfflineIMAP list pointed me to Twisted,
which I observe already contains an IMAP library that looks nice, plus
a different paradigm that may spare me from the whole thread groups
question altogether.

My questions, then, are these:

1. How does Twisted interact with things like threads and connection
   pooling?
   
   I currently let the user control the maximum number of connections
   that are open to a given server, and whenever the code needs to
   access an IMAP server, it grabs a connection from the appropriate
   pool and uses it (using threading primitives to block if no
   connection is available.)  Can Twisted handle multiple connections
   to multiple servers doing multiple things at once?  Can it do
   connection pools?  If it needs threading to do these things, does
   it play nice with it?

3. I notice the "Deferred" object and the benefits it provides when a
   server response is expected to take some time.  I assume that some
   sort of internal threading is taking place here?

   What if the server *request* takes some time -- say, uploading a
   2MB e-mail over a dialup link.  Are we still OK with handling that
   in a non-blocking fashion?

4. The howto "book" alludes to pending improvements on the mail
   infrastructure.  Anything I should be aware of here?

5. Is there any sort of unified mail API in Twisted (like JavaMail)
   that would present me with a single API to both IMAP and Maildir
   repositories, or is that something I need to do on my own?  (I've
   already done it, so it's no big deal to do that again)

6. OfflineIMAP supports several different user interfaces (two written
   with Tkinter, 1 using Curses, and three plain console ones.)  I would
   like to reimplement the Tkinter ones with wxPython, and am glad to
   see that Twisted works well with this.

   One concern, though, is that the users can supply a list of UIs to try
   in the config file: for instance, Tk, Curses, Noninteractive.  The
   system will try each one in turn until it reaches one that works.
   (For instance, Tk may not work if the user is not in X, and Curses
   may not work if stdout is not a tty).  It looks like this may be
   problematic with Twisted since the UI selection always seems to be
   known in advance with the import commands related to the reactor.
   Any thoughts here?

Thanks!

-- John Goerzen