Prompted by exarkun, I have put together some simple documentation for beginners starting with Twisted.<div><br></div><div>The points are made up of things I wish I had known from the start as a complete beginner.<br><div>
<br></div><div>It still needs lots of work. The layout needs to change, there are duplications and the points need to be made more succinctly.</div><div><br></div><div>But before I spend more time on honing the document, I thought it would be a good idea to get some feedback.</div>
<div><br></div><div>The information in the document may already exist and I have just overlooked it. </div><div><br></div><div>People may feel it is not appropriate for the documentation in Twisted and should go elsewhere.</div>
<div><br></div><div>Some of the information needs to be checked for accuracy and to avoid misleading readers.</div><div><br></div><div>Anyway I would appreciate any feedback positive or negative.</div><div><br></div><div>
John Aherne</div><div><br></div><div>Here ie the text:</div><div><div><br></div><div>Basic Information for anyone starting with Twisted</div><div><br></div><div>If you don't know mauch about TCP, then bear this in mind.</div>
<div><br></div><div>BASIC TCP</div><div><br></div><div>TCP is a stream of data. Once the connection is open, it stays open until closed.</div><div>It does not have a beginning or an end. It does not know about your messages.</div>
<div>You cannot wait until it sends your message since it will not tell you the message</div><div>has been sent. If you want to know if your message reached the other end, you need</div><div>to have in place a protocol for each end to respond that it received the data.</div>
<div>You will then need to implement a timeout for when there are problems otherwise</div><div>you may wait a long time for a response. TCP will wait forever. But intermediate</div><div>routers may time you out after 2 - 30 minutes if there is no traffic on the port. </div>
<div><br></div><div>TCP is just a stream of data. You have to process the stream of data looking for</div><div>a marker your application has placed that signifies the end of message to your</div><div>end of the application program.</div>
<div><br></div><div>Twisted provide the linereceiver and sendline functions to help in the common</div><div>case of using CR/LF as a terminator of messages, expecially for chat type</div><div>protocols and http.</div><div>
<br></div><div>The reactor and the select command will process the outgoing and incoming</div><div>buffers without blocking.</div><div><br></div><div>The reactor uses the select command. Each time the reactor cycles around, it will</div>
<div>use select to check the read and write buffers to see if any buffer is ready to</div><div>read or write. It will process those that are ready. Ignoring any not yet ready.</div><div><br></div><div>Anyone familiar with networking and select will probably already understand this.</div>
<div>Anyone not familiar will not realise it and needs to become familiar with how</div><div>select works.</div><div><br></div><div>If you want to know how Twisted processes network traffic, you should read up</div><div>on the select command.</div>
<div><br></div><div> </div><div>TWISTED - DIRECT SEND DATA CALLS </div><div> </div><div>For simple network activity you do not need to use deferreds. They are not</div><div>necessary. And you can get a lot done without deferreds just by using the</div>
<div>transport.write or sendline functions. This is shown in the simple Chat Server</div><div>example following.</div><div><br></div><div>Provided you are dealing in small amounts of data you will not block the reactor.</div>
<div>If you are sending megabytes of data in a file, that is a different</div><div>matter.</div><div><br></div><div>Using sendline directly is faster than using a deferred.</div><div><br></div><div>John Goerzen in his Apress book Python Network Fundamentals has a very simple</div>
<div>chat server example.</div><div><br></div><div><br></div><div>WHAT IS BLOCKING CODE</div><div><br></div><div>Blocking code is code that will block or may potentially block the continued</div><div>execution of the main reactor thread. Think for the most part of long running</div>
<div>processes or operations that may be long running, doing file or network i/o,</div><div>calculating cpu intensive work, operations that may timeout like doing a remote</div><div>call to another process or host machine, database operations are usually a</div>
<div>culprit, that may be flooded with work or crashed, the examples go on but are</div><div>mainly about i/o and cpu intensive operations.</div><div><br></div><div>When these things happen on the reactor / main thread they block the server from</div>
<div>doing anything else, it can't accept new connections, it can't do anything else</div><div>until this blocking activity has completed and returned control to the reactor</div><div>thread.</div><div><br></div><div>
WHAT ARE DEFERREDS</div><div><br></div><div>By and large they seem very similar to callbacks. They aren't, but seem to</div><div>perform the same sort of function. Please refer to other documentation on defers</div><div>
for more detailed explanation.</div><div><br></div><div>As everyone hears interminably on the twisted list, deferreds do not make </div><div>blocking code non-blocking. We all try it - but you shouldn't.</div><div><br>
</div><div>If you have blocking code, then first think about putting it into deferToThread</div><div>which will run the code in its own thread. It's not the only thing you can do</div><div>but it is a good start.</div>
<div><br></div><div>Return a deferred when setting up this threaded function and add appropriate</div><div>callbacks and errbacks. This will run the blocking code in its own thread.</div><div>You should not call transport.write or sendline functions directly from the</div>
<div>thread since this is not thread-safe.</div><div><br></div><div>In the thread you must call the callback or errback to return processing to the</div><div>reactor thread and then send any data from the reactor thread.</div>
<div><br></div><div>You can handle this without deferToThread by breaking the blocking code up into</div><div>smaller pieces. Sometimes you need to transfer a large file to a socket,</div><div>instead of trying to send it all at once send 10KB at a time and yield back to</div>
<div>the reactor and reschedule the next 10KB until finished. This will work, it might</div><div>not be the fastest way and still may block for an unacceptable amount of time</div><div>on just 10KB, depending on how heavily taxed the i/o system is at the moment.</div>
<div><br></div><div>Usually deferToThread is just easier to implement. </div><div><br></div><div>DATABASE PROCESSING TENDS TO BE BLOCKING</div><div><br></div><div>The adbapi module seems to be a good example of using deferreds and threads. The</div>
<div>adbapi module returns a deferred it has created, you add your callbacks to it.</div><div>The thread then calls your callback when ready. It does seem like the examplar</div><div>for doing deferreds.</div><div><br></div>
<div>The db stuff will normally block so put it in a thread and use deferreds to wait</div><div>the result or failure.</div><div><br></div><div>THREADS</div><div><br></div><div>twisted is meant to avoid the problems of using threads for network processing.</div>
<div>So why are we using threads. It's a way of moving potentially blocking code out</div><div>of the way so it avoids hanging the reactor.</div><div><br></div><div>THREADS WON'T NECESSARILY PREVENT BLOCKING</div>
<div><br></div><div>A point about the db calls is that they can be very intensive. If you need to </div><div>run some db function every 30 secs or 60 secs and the db takes 50% or more of the</div><div>time to generate the results, you won't have much time to service any incoming </div>
<div>requests that want to get results. The remote connections will be failing bigtime.</div><div><br></div><div>So then I suppose you should break the code into 2 programs. One that does the </div><div>db stuff, the other to handle the remote connections. The db code when it has </div>
<div>a result will then connect to the other program and pass across its results. </div><div>There may be better ways of doing this of course depending on circumstances.</div><div><br></div><div>WHEN TO USE DEFERREDS</div>
<div><br></div><div>If you have a cpu intensive process, then in all probability it will block the</div><div>reactor since it will take 100% cpu time while running - whether in the main thread</div><div>or in a separate thread. These are not good for running in twisted.</div>
<div><br></div><div>If you have I/O activity, such as reading lines of text from a disk file,</div><div>this seems a good candidate for deferreds.</div><div><br></div><div>This is what the dbapi module does. So it seems like a good example to follow.</div>
<div><br></div><div>As a general rule, it is simplest to use deferreds with threads. This is not</div><div>always true so circumstances may indicate a better way of running a deferred.</div><div><br></div><div>You still need to make sure that the bulk of the time is available for handling</div>
<div>connections. Otherwise you will start to have failing connections</div><div><br></div><div>Using sendline directly is faster than putting a deferred in between.</div><div><br></div><div>BEWARE WHEN USING DEFERREDS IN THREADS</div>
<div><br></div><div>Since deferToThread runs the function you pass to it in a non-reactor </div><div>thread, you may not use any non-thread-safe Twisted APIs in the function you </div><div>pass to it.</div><div><br></div>
<div>Beware of using shared data when running in the thread such as lists and dictionaries.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>
<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><br></div>
<div><br></div></div>