[Twisted-Python] deferred graph?

Eric Mangold teratorn at gmail.com
Sun Jul 22 10:55:26 MDT 2012


On Mon, 02 Jul 2012 19:31:14 -0500, Dan Stromberg <drsalists at gmail.com>  
wrote:

>
> On Thu, Jun 28, 2012 at 9:40 PM, Dan Stromberg <drsalists at gmail.com>  
> wrote:
>>
>> Hi.
>>
>> I'm a Twisted neophyte, but I've been using Python a long time.
>>
>> My question:
>> Is there a way of producing a deferred graph in a Python program at a  
>> given point in time?  Perhaps something based on graphviz and  
>> >>objgraph.py?  We're able to detect when we're having the problem, we  
>> just don't (yet) know its cause.
>>
>>
>> Background:
>> I'm looking at some code with a bit over 200  
>> addCallback/addErrback/addCallbacks in it.
>>
>> It's got a problem where sometimes the deferreds seem to just stop  
>> executing.  I see in the doc it says that this can happen if you've  
>> >>neglected to add an errback to the end of one or more of your  
>> deferred chains.
>>
>> One of the people who's been here longer than me, indicated that he's  
>> gone through the code looking for such issues, and didn't find >>any.   
>> This suggests to me that either there's some other cause, or that it  
>> really is a deferred without a final errback, but it's hidden in a  
>> >>dark corner of the code somewhere.
>>
>>
>> Thanks!
>>
> I put together something to do a single-point-in-time graph (as  
> distorted/delayed by the time it takes to go through all the objects in  
> the python >interpreter) of deferreds.  Please find it below as a shar  
> archive (wow, someone still does those?   Yeah, sometimes)

Cool stuff - but could you use an ordinary archive format? Asking the  
whole mailing list to run gobs of shell code is.... I dunno. not something  
I'm going to bother with even if I would like to.

-E


>  Anyway, given a tree of deferreds, this will produce a whatever.dot  
> file, which can be fed to graphviz' "dot -Tpdf whatever.dot >  
> whatever.pdf", >and the pdf has a rather nice-looking graph of the  
> deferreds.
>
> #!/bin/sh
> # This is a shell archive (produced by GNU sharutils 4.11.1).
> # To extract the files from this archive, save it to some FILE, remove
> # everything before the `#!/bin/sh' line above, then type `sh FILE'.
> #
> lock_dir=_sh32156
> # Made on 2012-07-02 21:48 UTC by <stromberg at aw50>.
> # Source directory was `/home/stromberg/src/twisted-experiments'.
> #
> # Existing files will *not* be overwritten, unless `-c' is specified.
> #
> # This shar contains:
> # length mode       name
> # ------ ---------- ------------------------------------------
> #   2181 -rwxr-xr-x deferreddump.py
> #   3174 -rwxr-xr-x wedgedump
> #
> MD5SUM=${MD5SUM-md5sum}
> f=`${MD5SUM} --version | egrep '^md5sum .*(core|text)utils'`
> test -n "${f}" && md5check=true || md5check=false
> ${md5check} || \
>  echo 'Note: not verifying md5sums.  Consider installing GNU coreutils.'
> if test "X$1" = "X-c"
> then keep_file=''
> else keep_file=true
> fi
> echo=echo
> save_IFS="${IFS}"
> IFS="${IFS}:"
> gettext_dir=
> locale_dir=
> set_echo=false
>
> for dir in $PATH
> do
>  if test -f $dir/gettext \
>     && ($dir/gettext --version >/dev/null 2>&1)
>  then
>    case `$dir/gettext --version 2>&1 | sed 1q` in
>      *GNU*) gettext_dir=$dir
>      set_echo=true
>      break ;;
>    esac
>  fi
> done
>
> if ${set_echo}
> then
>  set_echo=false
>  for dir in $PATH
>  do
>    if test -f $dir/shar \
>       && ($dir/shar --print-text-domain-dir >/dev/null 2>&1)
>    then
>      locale_dir=`$dir/shar --print-text-domain-dir`
>      set_echo=true
>      break
>    fi
>  done
>
>  if ${set_echo}
>  then
>    TEXTDOMAINDIR=$locale_dir
>    export TEXTDOMAINDIR
>    TEXTDOMAIN=sharutils
>    export TEXTDOMAIN
>    echo="$gettext_dir/gettext -s"
>  fi
> fi
> IFS="$save_IFS"
> if (echo "testing\c"; echo 1,2,3) | grep c >/dev/null
> then if (echo -n test; echo 1,2,3) | grep n >/dev/null
>     then shar_n= shar_c='
> '
>     else shar_n=-n shar_c= ; fi
> else shar_n= shar_c='\c' ; fi
> f=shar-touch.$$
> st1=200112312359.59
> st2=123123592001.59
> st2tr=123123592001.5 # old SysV 14-char limit
> st3=1231235901
>
> if touch -am -t ${st1} ${f} >/dev/null 2>&1 && \
>   test ! -f ${st1} && test -f ${f}; then
>  shar_touch='touch -am -t $1$2$3$4$5$6.$7 "$8"'
>
> elif touch -am ${st2} ${f} >/dev/null 2>&1 && \
>   test ! -f ${st2} && test ! -f ${st2tr} && test -f ${f}; then
>  shar_touch='touch -am $3$4$5$6$1$2.$7 "$8"'
>
> elif touch -am ${st3} ${f} >/dev/null 2>&1 && \
>   test ! -f ${st3} && test -f ${f}; then
>  shar_touch='touch -am $3$4$5$6$2 "$8"'
>
> else
>  shar_touch=:
>  echo
>  ${echo} 'WARNING: not restoring timestamps.  Consider getting and
> installing GNU `touch'\'', distributed in GNU coreutils...'
>  echo
> fi
> rm -f ${st1} ${st2} ${st2tr} ${st3} ${f}
> #
> if test ! -d ${lock_dir} ; then :
> else ${echo} "lock directory ${lock_dir} exists"
>     exit 1
> fi
> if mkdir ${lock_dir}
> then ${echo} "x - created lock directory ${lock_dir}."
> else ${echo} "x - failed to create lock directory ${lock_dir}."
>     exit 1
> fi
> # ============= deferreddump.py ==============
> if test -n "${keep_file}" && test -f 'deferreddump.py'
> then
> ${echo} "x - SKIPPING deferreddump.py (file already exists)"
> else
> ${echo} "x - extracting deferreddump.py (text)"
>  sed 's/^X//' << 'SHAR_EOF' > 'deferreddump.py' &&
> #!/usr/bin/python
> X
> # See also: http://twistedmatrix.com/trac/ticket/3858
> # and:      http://twistedmatrix.com/trac/ticket/1402
> # and:
> #   twisted.internet.defer also exposes a setDebugging(bool) function to
> #   store call stacks from creation and invocation in deferred objects,
> #   it's what I was using to try and figure out what was going on, but
> #   it's not nearly as at-a-glance as a graph is.
> X
> import gc
> import sys
> X
> import twisted.internet.defer
> X
> def escape(callback_type, obj):
> X    dummy = callback_type
> X    return '%s' % (str(obj).replace(' ', '_').replace(':',  
> '').replace('<', '').replace('>', ''), )
> X
> def dump(outfile = sys.stdout):
> X    outfile.write('digraph deferred_digraph {\n')
> X    for obj in gc.get_objects():
> X        if isinstance(obj, twisted.internet.defer.Deferred):
> X            len_callbacks = len(obj.callbacks)
> X            if obj.callbacks:
> X                outfile.write('\t%s -> %s\n' % (escape('', obj),  
> escape('', obj.callbacks[0][0][0])))
> X                outfile.write('\t%s -> %s\n' % (escape('', obj),  
> escape('', obj.callbacks[0][1][0])))
> X            for callbackpairno in range(len_callbacks - 1):
> X                current_callback = obj.callbacks[callbackpairno][0]
> X                current_errback = obj.callbacks[callbackpairno][1]
> X                current_callback_desc = current_callback[0]
> X                current_errback_desc = current_errback[0]
> X
> X                next_callback = obj.callbacks[callbackpairno + 1][0]
> X                next_errback = obj.callbacks[callbackpairno + 1][1]
> X                next_callback_desc = next_callback[0]
> X                next_errback_desc = next_errback[0]
> X
> X                outfile.write('\t%s -> %s;\n' % (escape('callback',  
> current_callback_desc), escape('callback', next_callback_desc)))
> X                outfile.write('\t%s -> %s;\n' % (escape('errback',  
> current_errback_desc), escape('errback', next_errback_desc)))
> X
> X                outfile.write('\t%s -> %s;\n' % (escape('callback',  
> current_callback_desc), escape('errback', next_errback_desc)))
> X                outfile.write('\t%s -> %s;\n' % (escape('errback',  
> current_errback_desc), escape('callback', next_callback_desc)))
> X
> X    outfile.write('}\n')
> X
> X
> SHAR_EOF
>  (set 20 12 06 29 18 02 00 'deferreddump.py'
>   eval "${shar_touch}") && \
>  chmod 0755 'deferreddump.py'
> if test $? -ne 0
> then ${echo} "restore of deferreddump.py failed"
> fi
>  if ${md5check}
>  then (
>       ${MD5SUM} -c >/dev/null 2>&1 || ${echo} 'deferreddump.py': 'MD5  
> check failed'
>       ) << \SHAR_EOF
> 45b2aed37cc2e57bc58f46a42eb2a2d4  deferreddump.py
> SHAR_EOF
>  else
> test `LC_ALL=C wc -c < 'deferreddump.py'` -ne 2181 && \
>  ${echo} "restoration warning:  size of 'deferreddump.py' is not 2181"
>  fi
> fi
> # ============= wedgedump ==============
> if test -n "${keep_file}" && test -f 'wedgedump'
> then
> ${echo} "x - SKIPPING wedgedump (file already exists)"
> else
> ${echo} "x - extracting wedgedump (text)"
>  sed 's/^X//' << 'SHAR_EOF' > 'wedgedump' &&
> #!/usr/bin/python
> X
> # Unhandled Errors
> X
> # From http://twistedmatrix.com/documents/current/core/howto/defer.html :
> X
> #   If a Deferred is garbage-collected with an unhandled error (i.e. it
> #   would call the next errback if there was one), then Twisted will  
> write
> #   the error's traceback to the log file. This means that you can
> #   typically get away with not adding errbacks and still get errors
> #   logged. Be careful though; if you keep a reference to the Deferred
> #   around, preventing it from being garbage-collected, then you may  
> never
> #   see the error (and your callbacks will mysteriously seem to have  
> never
> #   been called). If unsure, you should explicitly add an errback after
> #   your callbacks, even if all you do is:
> X
> #       # Make sure errors get logged
> #       from twisted.python import log
> #       d.addErrback(log.err)
> X
> import sys
> #mport time
> X
> import twisted.internet.defer
> import twisted.python
> X
> import deferreddump
> X
> def get_deferred():
> X    return twisted.internet.defer.succeed(1)
> X
> def cb1(value):
> X    print value
> X    return value * 2
> X
> def cb2(value):
> X    #raise ValueError
> X    print value
> X    return value * 2
> X
> def cb3(value):
> X    print value
> X    return value * 2
> X
> def cb4(value):
> X    print value
> X    return value * 2
> X
> def cb5(value):
> X    print value
> X    return value * 2
> X
> def cb6(value):
> X    print value
> X    return value * 2
> X
> def cb7(value):
> X    print value
> X    return value * 2
> X
> def cb8(value):
> X    print value
> X    return value * 2
> X
> def eb1(value):
> X    twisted.python.log.err()
> X
> def eb2(value):
> X    twisted.python.log.err()
> X
> def eb3(value):
> X    twisted.python.log.err()
> X
> def eb4(value):
> X    twisted.python.log.err()
> X
> def eb5(value):
> X    twisted.python.log.err()
> X
> def eb6(value):
> X    twisted.python.log.err()
> X
> def eb7(value):
> X    twisted.python.log.err()
> X
> def eb8(value):
> X    twisted.python.log.err()
> X
> def arbitrary_function(add_final_log):
> X    deferred = get_deferred()
> X
> X    # apparently we can use the same callback multiple times in the  
> same deferred
> X    first_deferred = deferred.addCallback(cb1)
> X    first_deferred.pause()
> X    deferred.addCallbacks(cb1, eb1)
> X    deferred.addCallbacks(cb2, eb2)
> X    deferred.addCallbacks(cb3, eb3)
> X    deferred.addCallbacks(cb4, eb4)
> X    deferred.addCallbacks(cb5, eb5)
> X    deferred.addCallbacks(cb6, eb6)
> X    deferred.addCallbacks(cb7, eb7)
> X
> X    with open('t.dot', 'w') as file_:
> X        deferreddump.dump(file_)
> X
> X    first_deferred.unpause()
> X    if add_final_log:
> X        deferred.addCallbacks(cb4, twisted.python.log.err)
> X    else:
> X        deferred.addCallback(cb4)
> X
> X    return deferred
> X
> def usage(retval):
> X    sys.stderr.write('Usage: %s --add-final-log\n' % sys.argv[0])
> X    sys.exit(retval)
> X
> def main():
> X    add_final_log = False
> X    while sys.argv[1:]:
> X        if sys.argv[1] == '--add-final-log':
> X            add_final_log = True
> X        elif sys.argv[1] in [ '-h', '--help' ]:
> X            usage(0)
> X        else:
> X            sys.stderr.write('%s: Unrecognized option: %s\n' %  
> (sys.argv[0], sys.argv[1]))
> X            usage(1)
> X        del sys.argv[1]
> X
> X    deferred = arbitrary_function(add_final_log)
> X    with open('/dev/null', 'w') as file_:
> X        file_.write(str(deferred))
> X
> X    sys.exit(0)
> X
> main()
> X
> SHAR_EOF
>  (set 20 12 07 02 21 46 02 'wedgedump'
>   eval "${shar_touch}") && \
>  chmod 0755 'wedgedump'
> if test $? -ne 0
> then ${echo} "restore of wedgedump failed"
> fi
>  if ${md5check}
>  then (
>       ${MD5SUM} -c >/dev/null 2>&1 || ${echo} 'wedgedump': 'MD5 check  
> failed'
>       ) << \SHAR_EOF
> d526664fcf63762f768303af91a74a5f  wedgedump
> SHAR_EOF
>  else
> test `LC_ALL=C wc -c < 'wedgedump'` -ne 3174 && \
>  ${echo} "restoration warning:  size of 'wedgedump' is not 3174"
>  fi
> fi
> if rm -fr ${lock_dir}
> then ${echo} "x - removed lock directory ${lock_dir}."
> else ${echo} "x - failed to remove lock directory ${lock_dir}."
>     exit 1
> fi
> exit 0
>



-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20120722/7032f6ac/attachment.html>


More information about the Twisted-Python mailing list