[Twisted-Python] txmongo vs. pymongo + callInThread

bret curtis psi29a at gmail.com
Mon Oct 12 01:51:36 MDT 2015


There is one thing that TxMongo does that PyMongo, Motor and the rest don't
do: handle deadlines and timeouts.

We're (HGST/WesternDigital) using MongoDB (with WiredTiger) in real-time
applications and one of our requirements was to honour application set
deadlines (or timeouts) on a per-call basis.

So if you are interested in that, we have it. :)

For those just tuning in...
Timeout: how many seconds the call is allowed run before raising an error.
Deadline: a time (unix timestamp) in the future that the call is allowed to
run before raising an error.

We take advantage of Twisted's native deferred cancellation mechanism in
addition to validation checks that no modifications happen to MongoDB when
a TimeExceeded is raised.

We've been very satisfied with how MongoDB has performed (post 3.0/WT)
along with TxMongo and Twisted.

Cheers,
Bret


On Fri, Oct 9, 2015 at 7:52 PM, Jonathan Vanasco <twisted-python at 2xlp.com>
wrote:

>
> On Oct 9, 2015, at 1:06 PM, Clayton Daley wrote:
>
> I (OP) was looking into these libraries/options as part of a side project
> that's been delayed by real work.  I'm also relegating Mongo to a caching
> layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB"
> and realizing I was going to have the exact same problems
>
> Thanks for the update!
>
> FWIW, (sorry, non-twisted!)  if you want to avoid mongodb:
>
> 1. We use redis for KV caching, and its really, really, really fast.
> 2. Riak can use LevelDB as a backing store -- leveldb supports snappy
> compression on documents
> 3. If you need the document search, i really like Elastic
>
> I don't really like/trust mongodb from past experience myself.  I would
> normally not use it for anything other than a read-through cache... except
> for my current need.  We have a web spider and need to archive the
> snapshots.  WireTiger, which was bought by mongodb and integrated into the
> 3.x branch, is the only storage backend I've found that will do "bucket
> level compression".  I forget if it's applied on pages or blocks, but it
> will basically apply zlib or snappy compression onto sets of documents.  I
> looked for weeks on something that could do that, it was the only option.
> We did a lot of storage tests on our typical data, and bucketed compression
> saves way more space than document compression or document compression
> using a seed.
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20151012/3ca9db9e/attachment-0002.html>


More information about the Twisted-Python mailing list