[Twisted-Python] Logging observers and threads

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Thu Jul 12 10:32:06 EDT 2012

On 01:56 pm, justin.venus at gmail.com wrote:
>For logging could we look at implementing posix aio on platforms that
>support it?  I would be willing to help code that.

As far as I know, all of the implementations of POSIX AIO are relatively 
low quality.  They include a number of tricky, low-level constraints 
(eg, char* to write must be page aligned) as well as some sad functional 
limitations such as blocking instead of being asynchronous if too many 
AIOs are started concurrently (with this limit applying to AIOs *system 
wide* in some cases).

If there are any good implementations, or prospects of thes 
implementations getting good sometime in the near future, it might be 
worth exploring, but I don't think it's something we can rely on now.

See also this stackoverflow question: http://bit.ly/NqWd5b

Corrections welcome, I haven't checked on the state of this in a few 

>On Jul 12, 2012 8:51 AM, <exarkun at twistedmatrix.com> wrote:
>>On 11:32 am, itamar at itamarst.org wrote:
>> >On 07/12/2012 05:17 AM, Tristan Seligmann wrote:
>> >>3) log observers should expect to be inovked in non-reactor threads,
>> >>making this a Mantissa bug.
>> >In theory this is the case - the logging howto says "The observer 
>> >to be thread safe if you anticipate using threads in your program." 
>> >API docs don't mention threads, though..
>> >
>> >In practice however, probably every custom observer gets this wrong, 
>> >we really need some infrastructure to make this easier or to fix it.
>> >Possibly we should have some way to indicate an observer is thread-
>> >safe,
>> >assume by default they are not, and if they are not wrap them in a
>> >reactor.callFromThread wrapper.
>>The logging system is already a bottleneck in some applications. 
>>even more work to try to make it properly thread-safe will probably 
>>it down even more.
>>I think it was a mistake to try to make it possible to use
>>twisted.python.log free-threaded.  I think we should think about 
>>rid of this claimed feature.  Perhaps it could be replaced with a more
>>explicit, probably more restricted, logging feature for non-reactor
>>Unless someone can magically fix the threading issues without hurting
>>performance.  Then, great, go for it.
>> >I would also like some infrastructure to make the file log observer 
>> >in its own thread, but that's almost the opposite use case :)
>>Sure, but that's probably based on top of an asynchronous file I/O API
>>that's not explicitly thread-based (but probably is thread-based in
>>practice, because what other kind of asynchronous file I/O even 
>>Also, to explicitly answer Tristan's question:
>>     It's a bug in the Mantissa log observer.
>>Twisted-Python mailing list
>>Twisted-Python at twistedmatrix.com

More information about the Twisted-Python mailing list