[Twisted-Python] First try at thread-safe FileDescriptor

Glyph Lefkowitz glyph at twistedmatrix.com
Fri Aug 10 18:49:04 EDT 2001


On Fri, 10 Aug 2001, Itamar wrote:

> First, my threaded server works great when using threaded mode.

HOORAY FOR TWISTED

> However, having a lock on every write call seems excessive, especially
> since some of my servers aren't threaded, so I tried rewriting
> abstract.FileDescriptor so that the write() and doWrite() methods are
> thread-safe without using locks.

This is an admirable goal.

However, from what I understand of Python's threading facilities, even
attribute access is somewhat suspect (as it may invoke a method call).  
At a somewhat fundamental level, what we are doing is manipulating an
internal data structure from one of several different threads.  There is
time spent within a system call that alters this data structure when the
main interpreter lock may not be held.

I think that even theoretically, this requires a lock.  (below I describe
an approach that may reduce the amount of locking...)

> 1) It doesn't actually work - if I turn of the threadble.synchronize()
> call at the end of the module then the server gets deadlocks.

Yeah, that's probably the biggest problem with this approach :)

> 2) How can I make the FileDescriptor backwards compatible? Add a custom
> __getstate__ that adds the unsentChunks list? If writeable 
> FileDescriptors are never pickled,
> which makes sense to me, then this is not issue.

They shouldn't ever be pickled.  In fact, I think that most of them mix in
styles.Ephemeral.

> 3) Will this slow things down?

Possibly, possibly not.  Difficult to say without profiling data (and
profiling MT applications is notoriously hard...)

> I may just give up and write a ThreadSafeFileDescriptor class,
> allowigng threaded servers to use it while non-threaded servers can
> use the regular FileDescriptor.

One alternative to reduce the number of locks and amount of locking going
on is to establish a different method for writing to a socket from a
thread.  Have the thread-safe version acquire a global lock on writing,
add some data (writer, string) to a global list, and relinquish it.  Then,
at the beginning of each select iteration, acquire that lock once: 

  lock.acquire()
  for writer, data in globalList:
    writer.write(data)
  lock.release()

and then keep all the actual write() calls happening in the one thread.

Another is to have the threaded implementation of write() actually call
socket.makefile('w').write(data); does anyone know the thread safety
implications of that?

                      ______      __   __  _____  _     _
                     |  ____ |      \_/   |_____] |_____|
                     |_____| |_____  |    |       |     |
                     @ t w i s t e d m a t r i x  . c o m
                     http://twistedmatrix.com/users/glyph







More information about the Twisted-Python mailing list