[Twisted-Python] possible error in twisted app
Jonathan Vanasco
twisted-python at 2xlp.com
Sun Jan 19 14:27:05 MST 2014
Thanks for the offer to help.
I was hoping someone would see an apparent bug in the outline, so i wouldn't have to build a a SSCCE
unfortunately, that wasn't going to fly, so I built out a self-contained version of the issue
Before sharing it, I added in some docs references to the example...
and then I noticed something peculiar, and seemed to have solved the problem !
the issue was this:
1. I used twisted.internet.defer.DeferredSemaphore to set up a semaphore service
2. I queued tasks with `semaphoreService.run( to_thread_function )`
3. `to_thread_function` ran a configurable method through `threads.deferToThread`
4. the configurable method ran something in twisted.enterprise.adbapi.ConnectionPool's `runInteraction`
when copying docs, i realized that I was running `toThread` and then `runInteraction` , which uses it's own thread. so i had threads spawning threads.
the base 'scaffold' for this daemon has been modified and patched since 2005 , so at some point i made an improvement and left some semi-functional legacy cruft in there.
i'm not sure of the specifics on how / why this manifests, but if I just use runInteraction and bypass using `deferToThread`, everything works out perfect.
if you're curious , i tossed the example online here
https://github.com/jvanasco/twisted_gist_001
`constants.py` has some toggles for playing with the return values of the various functions ( controls the base class and subclass )
it also lets you toggle to use the broken functionality ( thread within a thread ) or what seems to work fine now
happy i seemed to have solved this myself. still confused why the issue happened , but this fix ( only 1 thread ) seems to be fine and the more ideal approach
On Jan 18, 2014, at 7:17 PM, GMail wrote:
> Hi Jonathan,
>
> On Jan 17, 2014, at 6:22 PM, Jonathan Vanasco <twisted-python at 2xlp.com> wrote:
>
>> the following is a rough composite of what is going on. if anyone sees an obvious fix, i'd be greatly appreciative.
>
> I'd love to help, but a "rough composite" is hard to make guesses about, especially since you're talking about hard-to-predict memory-consumption behavior.
>
> Can you attach a <http://sscce.org/> which is actually runnable, for example, with a canned list of input URLs (or better yet with an included web server so the URLs can be localhost and more predictable), so we can debug and diagnose a running program instead of ideas about the outline of one?
>
> -glyph
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140119/acb3c186/attachment.html>
More information about the Twisted-Python
mailing list