<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Thanks for the offer to help.</div><div><br></div><div>I was hoping someone would see an apparent bug in the outline, so i wouldn't have to build a a SSCCE</div><div><br></div><div>unfortunately, that wasn't going to fly, so I built out a self-contained version of the issue </div><div><br></div><div>Before sharing it, I added in some docs references to the example...</div><div><br></div><div>and then I noticed something peculiar, and seemed to have solved the problem ! </div><div><br></div><div>the issue was this:</div><div><br></div><div>1. I used twisted.internet.defer.DeferredSemaphore to set up a semaphore service</div><div>2. I queued tasks with `semaphoreService.run( to_thread_function )`</div><div>3. `to_thread_function` ran a configurable method through `threads.deferToThread`</div><div>4. the configurable method ran something in twisted.enterprise.adbapi.ConnectionPool's `runInteraction`</div><div><br></div><div>when copying docs, i realized that I was running `toThread` and then `runInteraction` , which uses it's own thread. so i had threads spawning threads.</div><div><br></div><div>the base 'scaffold' for this daemon has been modified and patched since 2005 , so at some point i made an improvement and left some semi-functional legacy cruft in there.</div><div><br></div><div>i'm not sure of the specifics on how / why this manifests, but if I just use runInteraction and bypass using `deferToThread`, everything works out perfect.</div><div><br></div><div>if you're curious , i tossed the example online here</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> <a href="https://github.com/jvanasco/twisted_gist_001">https://github.com/jvanasco/twisted_gist_001</a></span></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>`constants.py` has some toggles for playing with the return values of the various functions ( controls the base class and subclass )</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>it also lets you toggle to use the broken functionality ( thread within a thread ) or what seems to work fine now</div><div><br></div><div><br></div><div>happy i seemed to have solved this myself. still confused why the issue happened , but this fix ( only 1 thread ) seems to be fine and the more ideal approach</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><div><div>On Jan 18, 2014, at 7:17 PM, GMail wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=us-ascii"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Jonathan,<br><br><div><div>On Jan 17, 2014, at 6:22 PM, Jonathan Vanasco <<a href="mailto:twisted-python@2xlp.com">twisted-python@2xlp.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">the following is a rough composite of what is going on. if anyone sees an obvious fix, i'd be greatly appreciative. </span><br style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></blockquote><br></div><div>I'd love to help, but a "rough composite" is hard to make guesses about, especially since you're talking about hard-to-predict memory-consumption behavior.</div><div><br></div><div>Can you attach a <<a href="http://sscce.org/">http://sscce.org/</a>> which is actually runnable, for example, with a canned list of input URLs (or better yet with an included web server so the URLs can be localhost and more predictable), so we can debug and diagnose a running program instead of ideas about the outline of one?</div><div><br></div><div>-glyph</div><br></div>_______________________________________________<br>Twisted-Python mailing list<br><a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python<br></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><br></div></div></div></span></div></span></div></span></div></span></div></span></span></div></body></html>