[Twisted-Python] Swapping a set of deferreds to diskOrganization: One Piece

[Vincent Bernat]

Hi !

I have  a function which returns a  deferred but will take  many time to
really  process an  item.  This  function  will be  feed with  a lot  of
items. For example,  the function could be called  1000 times per second
and will take  30 minutes to really process each  item. However, I don't
know the delay. The function could also process the items in real time.

When  the function  takes 30  minutes to  really process  the  item, the
reactor   will  keep   a  lot   of  deferreds   and  huge   portions  of
memory.  Therefore, I  would  like to  store  myself those  items (in  a
database for example)  and wait the function to be  ready to process all
the items (because the function knows when it will be ready).

I could store  the items directly in the database  and asks the function
an unique deferred  whose callback will be handled  when the function is
ready to process  my items. But since, the function  could be a function
that process  items in real  time (and more  faster than a  database), I
would like to use this "fallback" method only if the reactor has already
a  lot of deferred  for this  function. Ideally,  in this  case, current
deferred should be converted to database.

In short, I  would like to be able to  store/flush deferreds in database
when I have a  lot of them and keep them in memory if  I have only a few
of them.

For example,  a naive implementation  could be to implement  subclass of
Deferred and each Deferred instance can store itself to disk if a lot of
instances of  this subclass already exists.  This is not  exactly what I
want since I need a mechanism that will also flush all instances to disk
instead of  keeping them  in memory.  So an instance  should be  able to
replace a lot  of instances and recreate them  when the function becomes
ready.

Does such  a thing  already exists?  If it helps,  I can  guarantee that
function will process items in FIFO order.

Thanks.
-- 
I WILL NOT AIM FOR THE HEAD
I WILL NOT AIM FOR THE HEAD
I WILL NOT AIM FOR THE HEAD
-+- Bart Simpson on chalkboard in episode 8F13