wiki:Infrastructure/AdminScripts

Version 9 (modified by GaretJax, 21 months ago) (diff)

--

Note: this page is still under construction!

Fabric administration scripts

This page describes how to use the Fabric-based administration scripts to manage services running on twistedmatrix.com. The temporary location for the administration scripts is at: https://github.com/GaretJax/twisted-setup-temp

Getting the tools

The first step is to install the needed libraries and tools. I'm using virtualenv and virtualenvwrapper in this guide, you are free to install everything on your stock python.

mkvirtualenv env_name
workon env_name
pip install fabric
git clone git://github.com/GaretJax/twisted-setup-temp.git tmadmin  # May be svn or bzr as well, location is temporary
cd tmadmin

You can now list all available tasks by calling:

fab --list
Collection of utilities to automate the administration of Twisted's
infrastructure. Use this utility to install, update and start/stop/restart
services running on twistedmatrix.com.

Available commands:

    dns.install
    dns.restart
    dns.start
    dns.stop
    dns.update
    ...

Client configuration

Before being able to run any task, we need to configure the tool to work with Twisted's infrastructure and your login user. All configuration directives are contained in the config.py file. The base installation ships with a sample configuration file in config.py.sample. Copy it over and start editing it:

cp config.py.sample config.py
vim config.py

The two main directives to change are the HOSTS and USER ones. The first contains a list of hosts you want to administer, while the second one is, well, your ssh user to access the previously defined hosts. All found uppercase variables are lowercased and set as attributes of Fabric's env global variable. This means that any configuration directive recognized by Fabric can be defined in the config.py file as well.

Note: your user has to be at least in the service-admin group on the servers you want to manage in order to be able to start/stop/restart and update services. If you want to be able to install new services or give permissions to other users to manage them, your user has to be able to gain root access as well.

Server configuration conventions

Directory structure

Each service has its own directory under /srv. The structure of the service specific directory reflects the Filesystem Hierarchy Standard (e.g. pid files in /srv/<service>/var/run/<service>.pid, log files in /srv/<service>/var/log/<service>.log,...).

Users, groups and privileges

Each service runs as its own system user and owns his root directory (i.e. /srv/<service>). Each service user has to be part of the service group. To operate on a service (start/stop, update,...) a given user has to be part of the service-admin group. Each user in this group has permissions to sudo to every user in the service group.

As the service user is a system user, login for this user is disabled.

Init scripts

For each service, a System V style init script is provided in /srv/<service>/etc/init.d/<service>. This script is owned by the service user and has mode 0755. A symlink is provided in /etc/init.d, this allows the script to be managed by init(8) and allows updates to be applied to the service directory by a service administrator user.

For convenience, a runner which simplifies the creation of new init scripts is available at /usr/bin/twistd-service. A minimal example of an init script for an hypothetical DNS service is shown below:

#!/usr/bin/twistd-service

### BEGIN INIT INFO
# Provides:          dns
# Required-Start:    $named $network $time
# Required-Stop:     $named $network
# Should-Start:
# Should-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Twisted DNS server
# Description:       dns is a DNS server based on Twisted Names
### END INIT INFO

SERVICE='dns'
ARGS='dns'

This script is a complete System V compatible init script to run twistd from a specific virtualenv with the correct pid and logfile arguments under the correct user. It provides the start, stop, restart, status and zap commands. It can be run by the root user, the service specific user or any user in the service-admin group and it is started with the correct user.

Available tools

How to start/stop/restart services

Each service has its own Fabric namespace. Actions are available as part of each namespace. For example, the dns service can be started, stopped, and restarted as follows:

fab dns.start
fab dns.stop
fab dns.restart

How to update existing services

Similarly as done for managing the running states, an update task lives in each service namespace. It can be run as follows:

fab dns.update

Note that a restart is still needed after updating a service.

How to install new services

A service which was just added to the fabfile can be installed by running its install task:

fab dns.install

Note, however, that while the previous actions did not require root privileges (being member of the service-admin group was sufficient), installing a new service requires to be able to sudo to root. This is needed to create the necessary users, install additional packages and create the base environment.

How to give service administration privileges to another user

An existing user which does not currently have service administration tasks can be added to the service-admin group by executing the make_service_admin task and supplying a username:

fab make_service_admin:<username>

How to setup a new server from scratch

Adding additional tools or modifying existing ones