wiki:ContinuousIntegration/TestSlaveConfiguration

Version 17 (modified by thomasvs, 3 years ago) (diff)

add iptables hint

A build slave which is to run the Twisted test suite on a particular platform or in a particular configuration needs the following software:

  • Python
    • On Debian, the python-profiler package should also be installed.
    • On platforms where the Python headers are separated from the main Python package, these should also be installed (usually python-dev or python-devel).
  • A C compiler which can build Python extension modules (On POSIX, this usually means gcc; on Windows, some version of Visual Studio, depending on which version of Python is installed)
  • zope.interface (Debian package python-zope.interface)
  • On Windows, pywin32
  • Twisted - required by buildbot
  • BuildBot - only buildbot-slave is necessary
  • subversion
  • bazaar - sometimes packaged as bazaar-ng
  • bzr-svn
  • subvertpy
  • python-tdb

As well as all (or as many as possible) of the libraries Twisted depends on:

  • PyCrypto (Debian package python-crypto)
  • pyOpenSSL (depends on OpenSSL) (Debian package python-pyopenssl)
  • pyasn1 (Debian package python-pyasn1)
  • gmpy (depends on GMP) (Debian package python-gmpy)
  • SOAPpy (Debian package python-soappy)
  • pygtk (depends on gtk2) (Debian package python-gtk2)
  • subunit (Debian package python-subunit)

A build slave needs authentication credentials in order to be allowed to connect to the build master. Credentials can be obtained by emailing twisted dash python at twisted matrix dot com and expressing interest in running a slave.

Unfortunately a few tests in Twisted's test suite will fail if certain guarantees are not made by the environment. For example, the multicast tests try to use the actual network stack to send multicast packets. Here are some things that are known about this:

  • For the multicast tests to pass, multicast packets must be routable over localhost.
    • On OS X, this means "Internet Sharing" must be disabled
    • Some Linux distributions have a default iptables rule to drop multicast traffic, or require an explicit iptables rule to allow it. For example, on Red Hat:
      -A INPUT --dest 224.0.0.0/4 -j ACCEPT
      

A build slave is created with the buildbot create-slave command. This takes the master address, slave name, and slave password as arguments. The master address is buildbot.twistedmatrix.com:9987. Several other optional arguments are allowed. It is a good idea to specify a value of around 10 for --log-count to avoid unbounded log file growth. If there are special networking considerations, it may also be useful to specify a value for --keepalive. Other options generally aren't needed.

On the very first build by a new slave, a lengthy bzr initialization step is required. This usually needs to be completed manually after buildbot times out the operation for taking too long. The command which will fail on buildbot is:

bzr checkout svn://svn.twistedmatrix.com/svn/Twisted/trunk Twisted

with a working directory of /path/to/slave-root/builder-name/. The correct directory should have Twisted and .bzr directories in it already. Re-running the failed command in that directory will resume the checkout and eventually succeed (perhaps after quite a long time - dozens of minutes, at least - this is expected).