[Twisted-Python] possible error in twisted app

[Jonathan Vanasco]

Thanks for the offer to help.

I was hoping someone would see an apparent bug in the outline, so i wouldn't have to build a a SSCCE

unfortunately, that wasn't going to fly, so I built out a self-contained version of the issue 

Before sharing it, I added in some docs references to the example...

and then I noticed something peculiar, and seemed to have solved the problem !  

the issue was this:

1. I used twisted.internet.defer.DeferredSemaphore to set up a semaphore service
2. I queued tasks with `semaphoreService.run( to_thread_function )`
3. `to_thread_function` ran a configurable method through `threads.deferToThread`
4. the configurable method ran something in twisted.enterprise.adbapi.ConnectionPool's `runInteraction`

when copying docs, i realized that I was running `toThread` and then `runInteraction` , which uses it's own thread.  so i had threads spawning threads.

the base 'scaffold' for this daemon has been modified and patched since 2005 , so at some point i made an improvement and left some semi-functional legacy cruft in there.

i'm not sure of the specifics on how / why this manifests, but if I just use runInteraction and bypass using `deferToThread`, everything works out perfect.

if you're curious , i tossed the example online here

	https://github.com/jvanasco/twisted_gist_001

	`constants.py` has some toggles for playing with the return values of the various functions ( controls the base class and subclass )

	it also lets you toggle to use the broken functionality ( thread within a thread ) or what seems to work fine now


happy i seemed to have solved this myself.  still confused why the issue happened , but this fix ( only 1 thread ) seems to be fine and the more ideal approach








On Jan 18, 2014, at 7:17 PM, GMail wrote:

> Hi Jonathan,
> 
> On Jan 17, 2014, at 6:22 PM, Jonathan Vanasco <twisted-python at 2xlp.com> wrote:
> 
>> the following is a rough composite of what is going on.  if anyone sees an obvious fix, i'd be greatly appreciative.  
> 
> I'd love to help, but a "rough composite" is hard to make guesses about, especially since you're talking about hard-to-predict memory-consumption behavior.
> 
> Can you attach a <http://sscce.org/> which is actually runnable, for example, with a canned list of input URLs (or better yet with an included web server so the URLs can be localhost and more predictable), so we can debug and diagnose a running program instead of ideas about the outline of one?
> 
> -glyph
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20140119/acb3c186/attachment-0002.html>