[Twisted-Python] First try at thread-safe FileDescriptor
Glyph Lefkowitz
glyph at twistedmatrix.com
Fri Aug 10 18:49:04 EDT 2001
On Fri, 10 Aug 2001, Itamar wrote:
> First, my threaded server works great when using threaded mode.
HOORAY FOR TWISTED
> However, having a lock on every write call seems excessive, especially
> since some of my servers aren't threaded, so I tried rewriting
> abstract.FileDescriptor so that the write() and doWrite() methods are
> thread-safe without using locks.
This is an admirable goal.
However, from what I understand of Python's threading facilities, even
attribute access is somewhat suspect (as it may invoke a method call).
At a somewhat fundamental level, what we are doing is manipulating an
internal data structure from one of several different threads. There is
time spent within a system call that alters this data structure when the
main interpreter lock may not be held.
I think that even theoretically, this requires a lock. (below I describe
an approach that may reduce the amount of locking...)
> 1) It doesn't actually work - if I turn of the threadble.synchronize()
> call at the end of the module then the server gets deadlocks.
Yeah, that's probably the biggest problem with this approach :)
> 2) How can I make the FileDescriptor backwards compatible? Add a custom
> __getstate__ that adds the unsentChunks list? If writeable
> FileDescriptors are never pickled,
> which makes sense to me, then this is not issue.
They shouldn't ever be pickled. In fact, I think that most of them mix in
styles.Ephemeral.
> 3) Will this slow things down?
Possibly, possibly not. Difficult to say without profiling data (and
profiling MT applications is notoriously hard...)
> I may just give up and write a ThreadSafeFileDescriptor class,
> allowigng threaded servers to use it while non-threaded servers can
> use the regular FileDescriptor.
One alternative to reduce the number of locks and amount of locking going
on is to establish a different method for writing to a socket from a
thread. Have the thread-safe version acquire a global lock on writing,
add some data (writer, string) to a global list, and relinquish it. Then,
at the beginning of each select iteration, acquire that lock once:
lock.acquire()
for writer, data in globalList:
writer.write(data)
lock.release()
and then keep all the actual write() calls happening in the one thread.
Another is to have the threaded implementation of write() actually call
socket.makefile('w').write(data); does anyone know the thread safety
implications of that?
______ __ __ _____ _ _
| ____ | \_/ |_____] |_____|
|_____| |_____ | | | |
@ t w i s t e d m a t r i x . c o m
http://twistedmatrix.com/users/glyph
More information about the Twisted-Python
mailing list