[Twisted-Python] PATCH: setattr problem in ThreadAttr

[Matthias Urlichs]

Hi,

Itamar Shtull-Trauring:
> I don't see why you need a thread for queueing - a simple list would do,
> or a class wrapping one. Or look at transport's producer/consumer
> interface if this is for a protocol. Adding to queue is appending to
> list, dequeuing is getting item from beginning of list.

Some sample code would be nice... or pointers to which modules to look at.

> If, and *only* if you need threads - i.e. you are doing either blocking
> operations or really long running calculations, there are some options
> for you in Twisted.
> 
Yes -- the source data are coming from a database (and the consumer may
have to do database look-ups too). Since the consumer can be slower and I
don't want in-core memory to grow indiscriminately, some sort of
thread-safe bounded queuing makes sense.

> We have a thread pool dispatcher that queues methods to be run in the
> thread pool, that covers some the use cases. If you want to use a queue
> I suggest just sticking to Python's Queue or the slighly nicer
> twisted.python.timeoutqueue.
> 
Unrelated nit: twisted.python.timeoutqueue lacks a pop() function with
timeout (the wait+pop combo is NOT thread safe: you might have multiple
consumers), and, frankly, the basic idea of timeout-waiting on a queue the
way this module does it is just plain ugly.

TODO: Rewrite the whole thing, using mxQueue for the actual data and
Twisted support for everything else -- I'd use a Deferred() for pulling
one item from the queue, for instance, but a "call this function 
whenever you have a new item" interface might make more sense with a
high-volume queue. (This function might then want to return a Deferred()
to indicate that it will be ready for the next item LATER.) Same for
pushing stuff into the queue. And all of that thread-safe.  ;-)

I think I'll do that, probably sometime next week, because I need it.

> In general using threads in Python is slow anyway do to the global
> interpreter lock, which means in most cases only one thread at a time is
> running anyway.

While that is certainly true, working with blocking external interfaces
more-or-less requires them.

> That being said, Queue.Queue is rather slow in and of
> itself, and can be implemented in a much faster way by using one lock,
> and use mxQueue.
> 
I'd have to reinvent all of Queue's thread-support features, though.
That might be a nice exercise, but I'll spend my time on required features
first.  ;-)

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/