[Twisted-Python] txmongo vs. pymongo + callInThread

[Jonathan Vanasco]

On Oct 9, 2015, at 1:06 PM, Clayton Daley wrote:
> I (OP) was looking into these libraries/options as part of a side project that's been delayed by real work.  I'm also relegating Mongo to a caching layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB" and realizing I was going to have the exact same problems
> 
Thanks for the update!  

FWIW, (sorry, non-twisted!)  if you want to avoid mongodb:

	1. We use redis for KV caching, and its really, really, really fast.
	2. Riak can use LevelDB as a backing store -- leveldb supports snappy compression on documents
	3. If you need the document search, i really like Elastic

I don't really like/trust mongodb from past experience myself.  I would normally not use it for anything other than a read-through cache... except for my current need.  We have a web spider and need to archive the snapshots.  WireTiger, which was bought by mongodb and integrated into the 3.x branch, is the only storage backend I've found that will do "bucket level compression".  I forget if it's applied on pages or blocks, but it will basically apply zlib or snappy compression onto sets of documents.  I looked for weeks on something that could do that, it was the only option.  We did a lot of storage tests on our typical data, and bucketed compression saves way more space than document compression or document compression using a seed.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20151009/b82757ba/attachment-0002.html>