[Twisted-Python] TestInternet2.testPickledTimer Failed

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Thu Aug 1 05:50:20 MDT 2013


On 05:46 am, kylerzhang11 at gmail.com wrote:
>Hi,
>
>I'm a Google Summer of Code intern working on "Deferred Cancellation"
>project. I'm recently working on adding cancellation support to
>twisted.internet.task.LoopingCall.
>
>However, after I added the canceller to LoopingCall.deferred,
>the twisted.test.test_application.TestInternet2.testPickledTimer failed 
>due
>to a PicklingError.
>
>My branch is loopingcall-deferred-cancellation-6656. Here is the diff 
>of my
>code: http://twistedmatrix.com/~diffresource.twistd/6656
>
>[snip]
  File "/usr/lib/python2.7/pickle.py", line 748, in save_global
>    (obj, module, name))
>pickle.PicklingError: Can't pickle <function <lambda> at 0x8f1fb8c>: 
>it's
>not found as twisted.internet.posixbase.<lambda>

Two things to notice about the previous line.  One, it is trying to 
pickle a function defined using a lambda expression.  Two, it is trying 
to pickle something from twisted.internet.posixbase - which probably 
means it's trying to pickle the reactor.

You can run trial with --debug and it will drop into pdb when it hits 
this error.  Then you can walk up and down the call stack and inspect 
the objects pickle is operating on.  You can get some idea of where 
things are going wrong this way.
>
>I thought the reason was the circular references. However I searched 
>about
>it and found that pickle could handle the circular reference cases. But 
>the
>only significant change is that after I added the canceller, there is a
>circular reference between LoopingCall and LoopingCall.deferred. So I 
>don't
>know what's the problem. How can I fix this?

There are two changes that seem like they could be relevant.

First, LoopingCall now keeps a reference to the Deferred returned by 
application code.  This means anything reachable from that Deferred is 
going to get pickled when LoopingCall is pickled.  This jumped out at me 
first, but I don't think it's actually causing the problem.

Second, there is now a reference from the Deferred returned by 
`LoopingCall` back to the `LoopingCall` instance - via the bound 
`_cancel` method.  `TimerService` holds on to a reference to this 
`Deferred`.

Of course, stepping back, it doesn't make any sense to pickle 
LoopingCall - it explicitly refers to the reactor, so it's never 
actually going to be pickleable.

I suggest you take a look at TimerService and figure out why pickling 
one of those ever tries to pickle a LoopingCall (take a look around 
`__getstate__` and `volatile`, I think that's where the problem is).  I 
think you'll find an existing bug that the unit test previously failed 
to reveal but which your changes have revealed.

Jean-Paul



More information about the Twisted-Python mailing list