<div dir="ltr">Hi!<div><br></div><div>I'm trying to implement graceful shutdown of a HTTP server and I am unsure of the preferred way of implementing it. </div><div><br></div><div>I'm attempting to do this by adding a system event trigger for "before" "shutdown" that should stop accepting new requests and wait for current ones to finish.</div>
<div><br></div><div>My problem is that though my requests has fired their notifyFinish() deferreds the data has not been written to the client. Calling reactor.getWriters() shows me that there are active writers who are probably writing/flushing data to the client.</div>
<div><br></div><div>What is the preferred way of waiting for the requests to finish and finish writing the response to the client?</div><div><br></div><div>Here is example code for my own attempt:</div><div><br></div><div>
<pre class="" style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12px;line-height:16px;width:744px;margin-top:0px;margin-bottom:0px;padding:0px;color:rgb(0,0,0)"><div class="" id="file-graceful-twisted-shutdown-py-LC1">
<pre class="" style="font-family:Consolas,'Liberation Mono',Courier,monospace;width:744px;margin-top:0px;margin-bottom:0px;padding:0px"><div class="" id="file-graceful-twisted-shutdown-py-LC1"><pre class="" style="font-family:Consolas,'Liberation Mono',Courier,monospace;width:744px;margin-top:0px;margin-bottom:0px;padding:0px">
<div class="" id="file-graceful-twisted-shutdown-py-LC1"><pre class="" style="font-family:Consolas,'Liberation Mono',Courier,monospace;width:744px;margin-top:0px;margin-bottom:0px;padding:0px"><div class="" id="file-graceful-twisted-shutdown-py-LC1">
<span class="">from</span> <span class="" style="color:rgb(85,85,85)">twisted.internet</span> <span class="">import</span> <span class="" style="color:rgb(51,51,51)">reactor</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">defer</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC2"><span class="">from</span> <span class="" style="color:rgb(85,85,85)">twisted.application</span> <span class="">import</span> <span class="" style="color:rgb(51,51,51)">internet</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">service</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC3"><span class="">from</span> <span class="" style="color:rgb(85,85,85)">twisted.web.server</span> <span class="">import</span> <span class="" style="color:rgb(51,51,51)">Site</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">NOT_DONE_YET</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC4"><span class="">from</span> <span class="" style="color:rgb(85,85,85)">twisted.web.resource</span> <span class="">import</span> <span class="" style="color:rgb(51,51,51)">Resource</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC5"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC6"><span class="" style="font-weight:bold">class</span> <span class="" style="color:rgb(68,85,136);font-weight:bold">SlowResource</span><span class="">(</span><span class="" style="color:rgb(51,51,51)">Resource</span><span class="">):</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC7"> <span class="" style="color:rgb(51,51,51)">isLeaf</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(153,153,153)">True</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC8"> <span class="" style="color:rgb(51,51,51)">waiting_requests</span> <span class="" style="font-weight:bold">=</span> <span class="">[]</span></div><div class="" id="file-graceful-twisted-shutdown-py-LC9">
</div><div class="" id="file-graceful-twisted-shutdown-py-LC10"> <span class="" style="font-weight:bold">def</span> <span class="" style="color:rgb(153,0,0);font-weight:bold">notify_no_more_waiting</span><span class="">(</span><span class="" style="color:rgb(153,153,153)">self</span><span class="">):</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC11"> <span class="" style="font-weight:bold">if</span> <span class="" style="font-weight:bold">not</span> <span class="" style="color:rgb(153,153,153)">self</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">waiting_requests</span><span class="">:</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC12"> <span class="" style="font-weight:bold">return</span> <span class="" style="color:rgb(51,51,51)">defer</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">succeed</span><span class="">(</span><span class="" style="color:rgb(153,153,153)">None</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC13"> <span class="" style="font-weight:bold">return</span> <span class="" style="color:rgb(51,51,51)">defer</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">gatherResults</span><span class="">(</span><span class="" style="color:rgb(153,153,153)">self</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">waiting_requests</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">consumeErrors</span><span class="" style="font-weight:bold">=</span><span class="" style="color:rgb(153,153,153)">True</span><span class="">)</span> \</div>
<div class="" id="file-graceful-twisted-shutdown-py-LC14"> <span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">addBoth</span><span class="">(</span><span class="" style="font-weight:bold">lambda</span> <span class="" style="color:rgb(51,51,51)">ign</span><span class="">:</span> <span class="" style="color:rgb(153,153,153)">None</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC15"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC16"> <span class="" style="font-weight:bold">def</span> <span class="" style="color:rgb(153,0,0);font-weight:bold">write_result</span><span class="">(</span><span class="" style="color:rgb(153,153,153)">self</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">request</span><span class="">):</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC17"> <span class="" style="color:rgb(51,51,51)">request</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">write</span><span class="">(</span><span class="" style="color:rgb(221,17,68)">'{}'</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC18"> <span class="" style="color:rgb(51,51,51)">request</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">finish</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC19"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC20"> <span class="" style="font-weight:bold">def</span> <span class="" style="color:rgb(153,0,0);font-weight:bold">render_GET</span><span class="">(</span><span class="" style="color:rgb(153,153,153)">self</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">request</span><span class="">):</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC21"> <span class="" style="color:rgb(51,51,51)">reactor</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">callLater</span><span class="">(</span><span class="" style="color:rgb(0,153,153)">5</span><span class="">,</span> <span class="" style="color:rgb(153,153,153)">self</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">write_result</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">request</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC22"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC23"> <span class="" style="color:rgb(51,51,51)">d</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(51,51,51)">request</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">notifyFinish</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC24"> <span class="" style="color:rgb(153,153,153)">self</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">waiting_requests</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">append</span><span class="">(</span><span class="" style="color:rgb(51,51,51)">d</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC25"> <span class="" style="color:rgb(51,51,51)">d</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">addBoth</span><span class="">(</span><span class="" style="font-weight:bold">lambda</span> <span class="" style="color:rgb(51,51,51)">ign</span><span class="">:</span> <span class="" style="color:rgb(153,153,153)">self</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">waiting_requests</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">remove</span><span class="">(</span><span class="" style="color:rgb(51,51,51)">d</span><span class="">))</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC26"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC27"> <span class="" style="font-weight:bold">return</span> <span class="" style="color:rgb(51,51,51)">NOT_DONE_YET</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC28"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC29"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC30"><span class="" style="color:rgb(51,51,51)">slow_resource</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(51,51,51)">SlowResource</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC31"><span class="" style="color:rgb(51,51,51)">site</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(51,51,51)">Site</span><span class="">(</span><span class="" style="color:rgb(51,51,51)">slow_resource</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC32"><span class="" style="color:rgb(51,51,51)">application</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(51,51,51)">service</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">Application</span><span class="">(</span><span class="" style="color:rgb(221,17,68)">"MyApp"</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC33"><span class="" style="color:rgb(51,51,51)">server</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(51,51,51)">internet</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">TCPServer</span><span class="">(</span><span class="" style="color:rgb(0,153,153)">8080</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">site</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC34"><span class="" style="color:rgb(51,51,51)">server</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">setServiceParent</span><span class="">(</span><span class="" style="color:rgb(51,51,51)">application</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC35"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC36"><span class="" style="font-weight:bold">def</span> <span class="" style="color:rgb(153,0,0);font-weight:bold">wait_for_writers</span><span class="">():</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC37"> <span class="" style="color:rgb(51,51,51)">d</span> <span class="" style="font-weight:bold">=</span> <span class="" style="color:rgb(51,51,51)">defer</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">Deferred</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC38"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC39"> <span class="" style="font-weight:bold">def</span> <span class="" style="color:rgb(153,0,0);font-weight:bold">check_writers</span><span class="">():</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC40"> <span class="" style="font-weight:bold">if</span> <span class="" style="color:rgb(0,134,179)">len</span><span class="">(</span><span class="" style="color:rgb(51,51,51)">reactor</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">getWriters</span><span class="">())</span> <span class="" style="font-weight:bold">></span> <span class="" style="color:rgb(0,153,153)">0</span><span class="">:</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC41"> <span class="" style="color:rgb(51,51,51)">reactor</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">callLater</span><span class="">(</span><span class="" style="color:rgb(0,153,153)">0.1</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">check_writers</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC42"> <span class="" style="font-weight:bold">else</span><span class="">:</span></div><div class="" id="file-graceful-twisted-shutdown-py-LC43"> <span class="" style="color:rgb(51,51,51)">d</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">callback</span><span class="">(</span><span class="" style="color:rgb(153,153,153)">None</span><span class="">)</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC44"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC45"> <span class="" style="color:rgb(51,51,51)">check_writers</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC46">
<span class="" style="font-weight:bold">return</span> <span class="" style="color:rgb(51,51,51)">d</span></div><div class="" id="file-graceful-twisted-shutdown-py-LC47"> </div><div class="" id="file-graceful-twisted-shutdown-py-LC48">
<span class="">@defer.inlineCallbacks</span></div><div class="" id="file-graceful-twisted-shutdown-py-LC49"><span class="" style="font-weight:bold">def</span> <span class="" style="color:rgb(153,0,0);font-weight:bold">graceful_shutdown</span><span class="">():</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC50"> <span class="" style="font-weight:bold">yield</span> <span class="" style="color:rgb(51,51,51)">server</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">stopService</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC51"> <span class="" style="font-weight:bold">yield</span> <span class="" style="color:rgb(51,51,51)">slow_resource</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">notify_no_more_waiting</span><span class="">()</span></div>
<div class="" id="file-graceful-twisted-shutdown-py-LC52"> <span class="" style="color:rgb(153,153,136);font-style:italic">#yield wait_for_writers()</span></div><div class="" id="file-graceful-twisted-shutdown-py-LC53">
</div><div class="" id="file-graceful-twisted-shutdown-py-LC54"><span class="" style="color:rgb(51,51,51)">reactor</span><span class="" style="font-weight:bold">.</span><span class="" style="color:rgb(51,51,51)">addSystemEventTrigger</span><span class="">(</span><span class="" style="color:rgb(221,17,68)">'before'</span><span class="">,</span> <span class="" style="color:rgb(221,17,68)">'shutdown'</span><span class="">,</span> <span class="" style="color:rgb(51,51,51)">graceful_shutdown</span><span class="">)</span></div>
</pre></div></pre></div></pre></div></pre><div><br></div><div>Uncommenting the yield in graceful_shutdown gives me my intended behaviour but I don't like the solution since I have no idea if the writers there have anything to do with writing the results. They could be any writer writing anything I assume.</div>
<div><br></div><div>Running this code with twistd -ny graceful.tac and curling <a href="http://localhost:8080">http://localhost:8080</a> and then issuing a SIGINT to the twistd process results in an curl: (52) Empty reply from server.</div>
<div><br></div>-- </div><div>/Jonas</div></div>