wiki:Specification/DeclarativeMonkeyPatching

Version 6 (modified by jml, 7 years ago) (diff)

--

Problem

When writing unit tests, it is often necessary to provide alternate implementations of APIs which are used by the code being tested. Relying on the real implementations of these APIs would expand the scope of the test such that it no longer tested only the relevant unit. Various techniques exist for inserting the alternate implementations into the runtime environment so they are used:

  • Parameterize the objects which provide the implementations, as to __init__ or as attributes on the object the methods of which are being tested.
  • Factor usage of these implementations into well-defined functions or methods and them override those to provide the alternates.
  • Replace names in the global scope of the functions being tested.
  • Replace attributes on module objects which are used by the functions being tested.

These all accomplish the desired goal. However, each requires a somewhat ad-hoc approach to the problem, and rarely is any of the resulting test fixture code reusable except across very similar test methods. The first involves expanding an API in a way often only exploited by test code. The second often requires the definition of a new subclass solely for the use of the test code. The third and fourth are less invasive, but require even more whitebox knowledge of the code being tested and are prone to unexpected failures when the implementation changes in an otherwise trivial or straightforward manner.

Solution

In Progress...

Trial should expose a declarative API for modifying a particular global scope during a particular test. The best form for such an API would be a function decorator, since that would preclude the need for matching bookends in setUp and tearDown.

Here's a sketch of how such a thing might work:

## mything.py ##
from foo import bar

def doIt():
    return bar.value() + 1

## test_mything.py ##

import mything
from twisted.trial.unittest import TestCase, mocked

class MockBar(object):
    def value(self):
        return 4

class MyThingTest(TestCase):
    def test_myGlobalThing(self):
        self.failUnless(mything.doIt(), 5)
        self.failUnless(self.mock['bar'].valueCalled)
    test_globalMyThing = mocked(foo, bar=MockBar)(test_myGlobalThing)

Here, the 'mocked' decorator takes a single positional argument indicating the target for temporary rebinding and **kw indicating the names to temporarily rebind.


This doesn't address the fact that the implementation being tested might change to use different globals without the test being updated. Also, the use of a decorator might be a good idea, but eliminating the need for setUp/tearDown isn't the primary motivating concern here.

-exarkun


I don't understand how that might be addressed. The implementation might always change to do something the test isn't testing, and then it's the implementor's responsibility to change the test, or the reviewer's responsibility to change it. The reason I suggest a decorator rather than setUp/tearDown is that I don't see setUp/tearDown as "declarative". You can imperatively mutate global state in setUp, and forget the imperative command in tearDown to reverse it, which silently corrupts global state in the test.

I also still don't understand why we're using a wiki page rather than a ticket, especially now that we've gone into discussion mode.

-glyph


  • Ditto on 'wiki vs ticket'.
  • A concrete improvement can be made without implementing a fully declarative interface.
  • teratorn has already implemented a thing called 'masker' (see source:sandbox/teratorn/masker.py). If it were made to be re-usable, then it could helpfully simplify many tests.
  • A feature like addCleanup can go a long way to solving the tearDown problem. Simply put, addCleanup registers functions to be called after tearDown completes.

-jml, 2007-04-24