[Twisted-Python] caching suggestions

[Voznesensky Vladimir]

Well, if you need a fast database, you can use in-memory
database like
http://www.ispras.ru/~knizhnik/fastdb.html
(it's written in C++ for C++, so you will need to develop 
a glue
layer from C to Python)
But note, that all of your data should fit in RAM in such 
case.
Also note, that embedded databases delivers some delays
during commits in calling threads, so you should handle 
that
issues too.

On Thu, 14 Jun 2007 15:29:08 -0400
  Jonathan Vanasco <twisted-python at 2xlp.com> wrote:
> note: this is less about twisted than a project i've 
>built on twisted  that is having some issues.  everyone 
>here is smart, so i thought  this would be a good list to 
>ask on.
> 
> i've got a small twisted daemon that is working as a 
>proxy for  hosting content that we store on amazon-s3
> long story short: archive to s3 (redundancy, cheap 
>storage), twisted  daemon on our network fetches to cache 
>& serve.  we have cheaper  bandwidth, plus do a lot of 
>file monitoring / name abstraction that  amazon's network 
>won't support.  currently there are 10k documents on  s3, 
>which are accessible via 150k+ 'keys'.   the cache 
>basiscally  proxies the right doc for each key
> 
> I'm running into 2 issues with it:
> 	short term- keys are mapped to s3 files via a ton of 
>'hints' that i  store in bdb after fetching from 
>postgresql.  the hints need to be  refreshed every 1-5 
>minutes or so.  does anyone have a good  suggestion on 
>how to do that?  i could do this really easily with 
> sqlite, but i need to use bdb -- as sqlite isn't nearly 
>fast enough;  while bdb is.
> 
> 	long term-    everything is fine for now -- we only 
>have about 3gb  of data.  but thats going to be growing 
>to about 20gb soon, and we  want to limit the cache to an 
>active 10gb using an Adaptive  Replacement Cache 
>algorithm with a bdb datastore.  has anyone done 
> something similar in twisted or python in general?
> 
> any input would be appreciated.  thanks!
> 
> // Jonathan Vanasco
> 
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>- - - - - - -  - - - - - - - - - - - - - - - - - - -
> |   CEO/Founder SyndiClick Networks
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>- - - - - - -  - - - - - - - - - - - - - - - - - - -
> |      FindMeOn.com - The cure for Multiple Web 
>Personality Disorder
> |      Web Identity Management and 3D Social Networking
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>- - - - - - -  - - - - - - - - - - - - - - - - - - -
> |      RoadSound.com - Tools For Bands, Stuff For Fans
> |      Collaborative Online Management And Syndication 
>Tools
> | - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>- - - - - - -  - - - - - - - - - - - - - - - - - - -
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python