[Twisted-Python] caching suggestions

Jonathan Vanasco twisted-python at 2xlp.com
Thu Jun 14 15:29:08 EDT 2007


note: this is less about twisted than a project i've built on twisted  
that is having some issues.  everyone here is smart, so i thought  
this would be a good list to ask on.

i've got a small twisted daemon that is working as a proxy for  
hosting content that we store on amazon-s3
long story short: archive to s3 (redundancy, cheap storage), twisted  
daemon on our network fetches to cache & serve.  we have cheaper  
bandwidth, plus do a lot of file monitoring / name abstraction that  
amazon's network won't support.  currently there are 10k documents on  
s3, which are accessible via 150k+ 'keys'.   the cache basiscally  
proxies the right doc for each key

I'm running into 2 issues with it:
	short term- keys are mapped to s3 files via a ton of 'hints' that i  
store in bdb after fetching from postgresql.  the hints need to be  
refreshed every 1-5 minutes or so.  does anyone have a good  
suggestion on how to do that?  i could do this really easily with  
sqlite, but i need to use bdb -- as sqlite isn't nearly fast enough;  
while bdb is.

	long term-    everything is fine for now -- we only have about 3gb  
of data.  but thats going to be growing to about 20gb soon, and we  
want to limit the cache to an active 10gb using an Adaptive  
Replacement Cache algorithm with a bdb datastore.  has anyone done  
something similar in twisted or python in general?

any input would be appreciated.  thanks!

// Jonathan Vanasco

| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - - - - -
|   CEO/Founder SyndiClick Networks
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - - - - -
|      FindMeOn.com - The cure for Multiple Web Personality Disorder
|      Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - - - - -
|      RoadSound.com - Tools For Bands, Stuff For Fans
|      Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - - - - -






More information about the Twisted-Python mailing list