[Twisted-Python] PATCH: setattr problem in ThreadAttr
smurf at noris.de
Thu Nov 14 01:02:21 EST 2002
> I don't see why you need a thread for queueing - a simple list would do,
> or a class wrapping one. Or look at transport's producer/consumer
> interface if this is for a protocol. Adding to queue is appending to
> list, dequeuing is getting item from beginning of list.
Some sample code would be nice... or pointers to which modules to look at.
> If, and *only* if you need threads - i.e. you are doing either blocking
> operations or really long running calculations, there are some options
> for you in Twisted.
Yes -- the source data are coming from a database (and the consumer may
have to do database look-ups too). Since the consumer can be slower and I
don't want in-core memory to grow indiscriminately, some sort of
thread-safe bounded queuing makes sense.
> We have a thread pool dispatcher that queues methods to be run in the
> thread pool, that covers some the use cases. If you want to use a queue
> I suggest just sticking to Python's Queue or the slighly nicer
Unrelated nit: twisted.python.timeoutqueue lacks a pop() function with
timeout (the wait+pop combo is NOT thread safe: you might have multiple
consumers), and, frankly, the basic idea of timeout-waiting on a queue the
way this module does it is just plain ugly.
TODO: Rewrite the whole thing, using mxQueue for the actual data and
Twisted support for everything else -- I'd use a Deferred() for pulling
one item from the queue, for instance, but a "call this function
whenever you have a new item" interface might make more sense with a
high-volume queue. (This function might then want to return a Deferred()
to indicate that it will be ready for the next item LATER.) Same for
pushing stuff into the queue. And all of that thread-safe. ;-)
I think I'll do that, probably sometime next week, because I need it.
> In general using threads in Python is slow anyway do to the global
> interpreter lock, which means in most cases only one thread at a time is
> running anyway.
While that is certainly true, working with blocking external interfaces
more-or-less requires them.
> That being said, Queue.Queue is rather slow in and of
> itself, and can be implemented in a much faster way by using one lock,
> and use mxQueue.
I'd have to reinvent all of Queue's thread-support features, though.
That might be a nice exercise, but I'll spend my time on required features
Matthias Urlichs | noris network AG | http://smurf.noris.de/
More information about the Twisted-Python