Opened 7 years ago

Last modified 7 years ago

#6495 enhancement new

Need better Twisted doc/howto explaining how to implement Half-Synch/Half-Async design pattern

Reported by: Craig Rodrigues Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Branch:
Author:

Description

In this coursera class:

"Pattern-Oriented Software Architectures for Concurrent and Networked Software" https://class.coursera.org/posa-001/class/index

There was a programming assignment where the students needed to implement the Half-Synch/Half-Async pattern.

This pattern is described in these papers:

(1) http://www.cs.wustl.edu/~schmidt/PDF/HS-HA.pdf (2) http://www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf

and in this book: (1) http://www.dre.vanderbilt.edu/~schmidt/POSA/POSA2/

When the students in the class tried to implement this pattern using Twisted, they found the documentation a bit unclear in certain areas. While this type of implementation may not be something that most Twisted developers use, it still would be nice to clarify the documentation/howtos in this area.

This comment ( https://class.coursera.org/posa-001/forum/thread?thread_id=981&post_id=3542#post-3542 ) from Erik Habbinga best summarizes the problems that a few of the students had with the documentation:

=================================================================================================================== I was trying to make the Python assignment match Professor Schmidt's C++ assignment as closely as possible. I'm interpreting the assignment to have two motives: achieve familiarity with the patterns, and achieve familiarity with the framework. I'll admit that doing the work for this assignment using a threadpool is overkill for the task at hand, but it does serve the purpose of demonstrating that twisted has threadpools available.

I agree that the twisted documentation is poor, and having a clear example of threadpool usage would be helpful. Check out twisted's test code for their threads implementation (http://twistedmatrix.com/trac/browser/trunk/twisted/test/test_threads.py), in particular the DeferToThreadPoolTestCase class for an example of creating/destroying a threadpool and using it to defer work. ===================================================================================================================

My recommendation is that we add a new example in the howto section which extends either the "Writing a TCP Server" or "The Evolution of Finger" example to implement the Half-Sync/Half-Async pattern.

This example could be really simple and just extend the "Echo" server example to call deferToThread inside the dataReceived() method of the server.

This example could illustrate the use of Threadpools and Deferreds, but the example should also explain some of the downsides of this design in terms of performance, ordering problems, etc.

Douglas Schmidt is teaching this course again next year in the spring of 2014. Aligning the Twisted docs with this course is a good opportunity to get newcomers who want to learn good design practices for networked software, to pick up Twisted, and see how well the designs are implemented in Twisted.

Change History (2)

comment:1 Changed 7 years ago by Craig Rodrigues

To clarify, this was the original assignment given in the Coursera course:

https://class.coursera.org/posa-001/human_grading/view/courses/970268/assessments/41/submissions

I am including the wording of the assignment below:

The purpose of this assignment is to deepen your understanding of the Half-Sync/Half-Async pattern in the context of Python and Twisted. In particular, you will write a platform-independent server program that builds upon the solution from PA#3 by using the Half-Sync/Half-Async pattern to accept a connection from one or more clients and echoes back what the client sent. In brackets below are some hints about what kinds of Twisted classes/methods you might want to look up to do these.

In addition to reusing the protocol.ServerFactory and protocol.Protocol classes from PA#3, you should also use the python.threadpool class for this assignment. The other hints are simply for your convenience.

Note: most code in twisted is not thread safe, so writing data to a transport from a thread needs to happen in the twisted reactor main loop (http://twistedmatrix.com/documents/11.0.0/core/howto/threading.html#auto0).

The server program should do the following activities:

Create an EchoServerHandler class that derives from twisted.internet.protocol.Protocol. This class will implement a few methods to implement the half-async and half-sync portions of the server: At initialization, create an instance of twisted.python.threadpool and start the threadpool Implement the dataReceived() hook method to receive client input and perform the "half-async" portion of the server. This method should read the client data until the end of a line is reached (i.e., the symbols "\n", "\r", or "\r\n" are read). This data is passed to a method EchoFromThread running on a thread from the threadpool, with a callback to another method ReturnFromThread. Note that implementing the half-async portion properly may require multiple trips through the reactor to read each chunk of client data via a single non-blocking recv() each time. [python.twisted.threads.deferToThreadPool] EchoFromThread will prepend the thread ID to incoming data and return the resulting string, which will be received by the ReturnFromThread callback. This method is performing the "half-sync" portion of the server [python.twisted threadable.getThreadID] ReturnFromThread will send the new data (including thread ID) to the client, running in the same context as the reactor main loop. At de-initialization, stop the threadpool Create an EchoServerFactory class that derives from twisted.internet.protocol.ServerFactory and implement its buildProtocol() hook method so that it returns a EchoServerHandler. Create a twisted.internet.reactor that registers a EchoServerFactory when a connection on the TCP port of your choice is received [listenTCP] and start that reactor. For this assignment, you can simply exit the server process when you're done, which will close down all the open sockets.

Make sure to clearly document in your solution which classes or methods play which roles in the Wrapper Facade, Reactor, Acceptor-Connector, and Half-Sync/Half-Async patterns.

This server program should be implemented in Python 2.6 or newer, utilizing the twisted reactor framework, and include at least one file (name it like server.txt so uploading is accepted on Coursera!) for your server which is the basis for the evaluation. An evaluator should be able to run your program with a command such as 'python server.txt' and your program should successfully run!

A second client.txt file may optionally be included for a simple console client if you write one during your solution development as a test program and wish to include it for the convenience of evaluators, but it is not required and will not be assessed as part of this exercise, just used as a test driver for your server. You can also use a telnet client (such as putty) as a test driver for your server.

Twisted classes of note: Reactor (IReactorTcp): http://twistedmatrix.com/documents/13.0.0/api/twisted.internet.interfaces.IReactorTCP.html ServerFactory: http://twistedmatrix.com/documents/13.0.0/api/twisted.internet.protocol.ServerFactory.html Protocol: http://twistedmatrix.com/documents/13.0.0/api/twisted.internet.protocol.Protocol.html Threadpool: http://twistedmatrix.com/documents/current/api/twisted.python.threadpool.html

Please add the Python version you used into a comment in source code.

Rubric

35 possible points

5 points – code is well-structured and design makes sense

20 points – Wrapper Facade, Reactor, Acceptor-Connector, and Half-Sync/Half-Async patterns are used correctly (and the classes in the solution are clearly mapped to the appropriate roles in the patterns)

5 points – code demonstrates the appropriate level of readability for others to understand the intent of the patterns and execution of the program

5 points – code compiles and runs smoothly

comment:2 Changed 7 years ago by Jean-Paul Calderone

Would a simpler way to phrase this enhancement be "Add a ThreadPool howto"?

Note: See TracTickets for help on using tickets.