Annvix
Personal tools



/User Guide/Services

From Annvix

(Redirected from Documentation/Services)

Managing Supervised Services

Unlike most Linux distributions, Annvix uses a different means of managing daemons and services. Those who have been using Linux for a while are likely familiar with initscripts; shell scripts that handle the starting and stopping of system services. These scripts are typically called as:

# /etc/init.d/script start

Or:

# service script start

These initscripts tend to have a few flaws, however. For one, they are big and bulky and, more often than not, unnecessarily bloated. This makes them slow to start services. They also inherit the system's environment which isn't a problem when they're starting from init, but problems may occur when administrators restart services as they can inherit variables from the current user's shell environment. Finally, if a service stops or is killed in some manner, the service stays down. This may be good if a particular service is being configured, but can be very problematic on systems that require high availability. For instance, assume an errant CGI script kills Apache (unlikely, but assume it could happen). If this happens, the httpd service stays down, denying legitimate requests.

The runit package, written by Gerrit Pape (and based on DJB's daemontools) solves these problems in a very elegant way. A single service, called runsvdir, watches all services started under it. These services all receive clean environments and use a single run script which handles starting the service. If a service goes down, runsvdir restarts the service within five seconds.

Another benefit of using runsvdir is at boot-up. When using traditional initscripts, the services are started one after another; if one service is having difficulty starting and takes time, it delays the entire boot process. With runsvdir, all services are started in parallel. If one service depends on another that is not yet available (and thus exits), runsvdir will restart it at which time the other service should be available. If not, runsvdir will continue to restart the service until it stays up (or indefinitely if there are problems).

That being said, supervise control can be somewhat cryptic so a wrapper was written to handle "talking" to services in a more familiar manner. The wrapper script is called srv.

Contents

Using srv

The srv package was originally a collection of scripts that included Bruce Guenter's supervise-scripts, as well as a "master" script developed specifically for Annvix called srv. Before 1.0-RELEASE, srv evolved into a single script that incorporated the functionality of the supervise-scripts into a single script. With 1.1-RELEASE, srv has been made more robust and feature-rich.

srv controls services in much the same way as service (used by initscripts), but it also provides the same functionality as chkconfig. Unlike many Linux distributions, if you install a daemon service on Annvix, it does not automatically start for you. You need to explicitly enable the service before it will ever start.

By default, openssh-server is installed on Annvix but, as with all daemon services, it is not started by default or even made available by default. To obtain a list of available services, execute:

# srv --list

service                   status   pid      started
-------------------------------------------------------------
crond                     up       3397     02/02/2007 12:03:45 AM
crond/log                 up       3366     02/02/2007 12:03:45 AM
mingetty-tty1             up       5785     02/02/2007 12:10:41 AM
mingetty-tty2             up       3302     02/02/2007 12:03:44 AM
mingetty-tty3             up       3399     02/02/2007 12:03:45 AM
mingetty-tty4             up       3332     02/02/2007 12:03:44 AM
mingetty-tty5             up       3331     02/02/2007 12:03:44 AM
mingetty-tty6             up       3362     02/02/2007 12:03:46 AM
socklog-klog              up       3409     02/02/2007 12:03:45 AM
socklog-klog/log          up       3408     02/02/2007 12:03:45 AM
socklog-rklog             -        -        -                   
socklog-rklog/log         -        -        -                   
socklog-unix              up       3444     02/02/2007 12:03:45 AM
socklog-unix/log          up       3443     02/02/2007 12:03:45 AM   
sshd                      -        -        -                   
sshd/log                  -        -        -   

As you can see here, only crond, socklog-klog, socklog-unix, and the mingetty services are available and running. These services are the only ones to start "out of the box". You can also see that sshd is installed, but is not available. To enable sshd, execute:

# srv --add sshd
Adding service 'sshd': .......done
# srv --list

service                   status   pid      started
-------------------------------------------------------------
crond                     up       3397     02/02/2007 12:03:45 AM
crond/log                 up       3366     02/02/2007 12:03:45 AM
mingetty-tty1             up       5785     02/02/2007 12:10:41 AM
mingetty-tty2             up       3302     02/02/2007 12:03:44 AM
mingetty-tty3             up       3399     02/02/2007 12:03:45 AM
mingetty-tty4             up       3332     02/02/2007 12:03:44 AM
mingetty-tty5             up       3331     02/02/2007 12:03:44 AM
mingetty-tty6             up       3362     02/02/2007 12:03:46 AM
socklog-klog              up       3409     02/02/2007 12:03:45 AM
socklog-klog/log          up       3408     02/02/2007 12:03:45 AM
socklog-rklog             -        -        -                   
socklog-rklog/log         -        -        -                   
socklog-unix              up       3444     02/02/2007 12:03:45 AM
socklog-unix/log          up       3443     02/02/2007 12:03:45 AM   
sshd                      up       5644     02/10/2007 10:25:02 AM
sshd/log                  up       9177     02/10/2007 10:25:55 AM

Now you can see that sshd is available and running.

Retaining Status Across Reboots

srv, and runsvdir, respect your decisions across a reboot. This means that if you turn off a service using:

# srv --down sshd

on the next reboot, sshd will stay down. You will need to explicitly start it again, so keep this in mind when you turn services on and off.

srv Options

The best source of information on srv is the manpage. However, the following summarizes all of the commands you can pass to srv:

  • --help - lists all available commands
  • --list | -l ([service]) / - lists all available services or just the named service
  • --info | -i [service] - displays verbose information on a single service
  • --add | -a [service] - adds a service to /service and starts it
  • --del | -z [service] - removes a service from /service, stopping it first
  • --up | -u [service] - starts the named service
  • --down | -d [service] - stops the named service; on a reboot it will remain down
  • --restart | -r [service] - stops then starts the named service
  • --reload | -h [service] - restarts a service by sending it a HUP signal

The --add and --del commands are similar to those from chkconfig. The --list command is similar to chkconfig --list.

The --up and --down commands work similar to the service commands "start" and "stop" for initscripts.

The --list command, as previously illustrated, will show what services are installed, which are available to be started, and what their status is by showing a "-" (available but not in /service), "up" (running), or "down" (not running).

The --info command gives extended status information on a particular service. For instance, to check the status of sshd, you would use:

# srv --info sshd
Service:        sshd
------------------------------------------------
Installed:      Thu 01 Feb 2007 11:19:45 PM MST
RPM:            openssh-server-4.5p1-6584avx
RPM Desc:       OpenSSH Secure Shell protocol server (sshd)
------------------------------------------------
Managed:        Yes
Status:         Up
Pid:            3376
Processes: 
                tcpsvd(3376)-+-sshd(24424)---sshd(24426)---sh(24427)---screen(24428)---zsh(24429)
                             `-sshd(8785)---sshd(8787)---zsh(8788)---su(8803)---bash(8813)
Started:        729012 seconds ago
When:           02/02/2007 12:03:46 AM
Is Logged:      Yes
Log Dir:        /var/log/service/sshd
Log Status:     Up
Log Pid:        3375
Log Started:    729013 seconds ago
Log When:       02/02/2007 12:03:45 AM

Dependencies:   sshd
------------------------------------------------
Requires:       <none>
Needed By:      <none>

As you can see, using runit makes sense in a number of areas, and the srv wrapper script makes it very easy to manage system daemons. Nearly every daemon shipped with Annvix uses srv.

If you wish to know a little bit more of how supervise works, see #How It Works. For those who just need to know how to turn things on and off, you now know.

Peers

srv also handles a concept called "peers", which is an access control mechanism similar to using /etc/hosts.allow and /etc/hosts.deny. Services that use tcpsvd make use of peers to allow or deny access to those services. tcpsvd is another "super-server" similar to xinetd. It merely answers any connection requests, determines if the connecting system is permitted to use the service, and if so, hands it off to the appropriate program.

For instance, the sshd service uses tcpsvd in exactly this way. It's run script looks like:

#!/bin/execlineb

/bin/fdmove -c 2 1

/bin/export PATH "/sbin:/bin:/usr/sbin:/usr/bin"

/sbin/chpst -e /etc/sysconfig/env/tcpsvd/
/sbin/chpst -e ./env/

/bin/multisubstitute {
    import -D "localhost" HOSTNAME
    import -D 0 IP
    import -D 22 PORT
    import -D 20 MAX_CONN
    import -D 5 MAX_PER_HOST
    import -D 20 MAX_BACKLOG
    import OPTIONS
}

/sbin/tcpsvd -v -l ${HOSTNAME} -x peers.cdb -c ${MAX_CONN} -C ${MAX_PER_HOST} -b ${MAX_BACKLOG} ${IP} ${PORT}
     /usr/sbin/sshd -i ${OPTIONS}

Essentially what happens here is that a number of variables are imported first from the global /etc/sysconfig/env/tcpsvd environment directory, then the service's own ./env environment directory. The important thing to note here is the -x peers.cdb argument to tcpsvd.

This argument instructs tcpsvd to use peers information for ACLs as found in the peers.cdb file for this service. This file is a CDB copy of the contents of the ./peers directory for this service. This file is constructed every time srv is used to manage peers information for the service, by first writing the peers information to ./peers and then creating the CDB file by using the ipsvd-cdb tool.

Structure of ./peers

The peers directory is a regular directory that contains IP addresses, partial IP addresses, or hostnames with certain permissions that tell tcpsvd whether or not that IP (or hostname) is allowed to connect to the service. The catch-all file is ./peers/0, and it's default permissions determine the default access type. This directory is essentially an ipsvd instructions directory; as such you can get more details from the ipsvd-instruct(5) manpage.

The file's permissions determines the access to the service for that IP or hostname:

  • If neither the user's read permission, nor the user's execute permission is set for the file, the connection is closed immediately
  • If the file has the user's execute permission set, ipsvd reads the contents of the file and runs /bin/sh -c '<contents>' instead of the default program given att he command line for this connection (i.e. the service itself)
  • If the file has the user's read permission set, ipsvd reads the contents of the file and interprets each line as an instruction for this connection; thus allowing the connection

Essentially, to deny access to a connection, chmod 0 the file. To allow access to a connection, chmod 0644 the file. If you want to get fancy, you can have the file contain an alternate program to execute as it's contents and chmod 0755 the file.

For instance, to disallow all connections to the sshd service, but allow connections from the 10.0.5.0 network and a single remote IP address (say 12.34.56.78), the ./peers directory would look thus:

# ls -l
total 4
---------- 1 root admin 0 Dec 29 16:09 0
-rw-r--r-- 1 root root  0 Feb  4 17:43 10.0.5
-rw-r--r-- 1 root root  2 Nov 18 07:28 12.34.56.78

How srv handles peers

Manually manipulating ./peers is fine, but srv makes it simple to do basic ACLs. It doesn't do things such as executing alternate programs based on the connection, but it will manipulate the peers information to the point that you can allow and deny access based on IP or hostname or network, and you can also set per-peer environment pair values, as well as the number of concurrent connections to the service.

The related srv commands are:

  • --peers | -p [service] - set peer information for the service (see below), or rebuild the peer instructions for the service if no peer control options are given
  • --env | -e [service] - set global environment pair values for the service

Along with the --peers command, you can use the following peer control options:

  • --grant | -g - grant access to this service to the specified hosts, IPs, or networks
  • --exclude | -x - deny access to this service to the specified hosts, IPs, or networks
  • --remove | -y - remove peer information for the specified hosts, IPs, or networks
  • --env | -e - set per-peer environment pair values
  • --max | -m - set the number of concurrent connections for the specifies hosts, IPs, or networks to this service

A quick example:

# srv --peers rsync -g 192.168.0.12 -e TRUSTED=yes -m 5

This command sets up peer information for the rsync service. It grants access to the IP 192.168.0.12, sets the environment variable $TRUSTED to "yes" (this is made available to the service), and sets the maximum concurrent connections to 5.

This command also illustrates that while you can have default "accept" to a service, you can also override certain configuration options based on connecting IP or hostname. In this case, if access to the rsync service was globally permitted, the maximum number of concurrent connections per IP might be 2 for every IP other than that specified, which would permit 5. Likewise, the environment variable $TRUSTED is set when this IP connects where it is not set for all others (perhaps rsync is configured to allow password-less connections if that variable is set, or it may permit access to other repositories, etc.).

As you can see, peers management is simple and very powerful. You can completely change how a service responds based on the connecting system.

Another example:

# chmod 0 ./peers/0
# srv --env sshd MAX_PER_HOST=1
# srv --peers sshd -g secure.host.com -e OPTIONS='-o "PermitRootLogin yes"'-m 10
# srv --peers sshd -g host.com -m 3

This series of commands does a few interesting things. The first is to disable all connections to sshd by default. The second is to set the maximum concurrency for all hosts to one connection per IP. The third is to grant access to sshd to secure.host.com with the environment variable $OPTIONS set to '-o "PermitRootLogin yes"' and a concurrency of 10 connections. The last is to raise the concurrency to three connections per IP to any *.host.com connections.

The third command is the most interesting here. Because the sshd service passes the value of $OPTIONS directly to sshd itself, you can directly manipulate the server based on the incoming connection. For instance, in this example, perhaps "PermitRootLogin" is set to "No" in sshd_config. For all connections this will be the case, however, if a connection comes from secure.host.com, all of a sudden sshd will allow root logins because that option is passed to it.

The resulting ./peers/secure.host.com file would look like:

C10
+  OPTIONS=-o "PermitRootLogin yes"

Finally, the use of host names is fine, but keep in mind that it must be successfully looked up via DNS first. This means that if the IP 12.34.56.78 does not have a reverse DNS entry to secure.host.com, access will be granted based on the IP address and not the hostname.

Also note that in order to make use of this, you will need to add -h to the tcpsvd command in the ./run script. By default, no services will attempt to do DNS lookups on incoming connections as it can get expensive in terms of connection speeds. To enable this, simply change ./run so that instead of looking like:

/sbin/tcpsvd -v -l ${HOSTNAME} -x peers.cdb -c ${MAX_CONN} -C ${MAX_PER_HOST} -b ${MAX_BACKLOG} ${IP} ${PORT}
     /usr/sbin/sshd -i ${OPTIONS}

it looks like:

/sbin/tcpsvd -v -h -l ${HOSTNAME} -x peers.cdb -c ${MAX_CONN} -C ${MAX_PER_HOST} -b ${MAX_BACKLOG} ${IP} ${PORT}
     /usr/sbin/sshd -i ${OPTIONS}

If you change a run script, keep in mind that upgrades to that package will never overwrite your changes. However, the new script will be named run.rpmnew and you will most likely want to examine the file for changes that may have been made and merge them in manually (Annvix treats a run script as a configuration file as far as rpm is concerned to allow users to customize run scripts to suit their needs without fear of their changes getting lost).

How It Works

Of course, this is all fascinating and if all you're doing is administering an Annvix system "as is", this is about all you need to know. However, if you wish to run other services supervised, you may wish to know how it all works.

At boot, Annvix uses runit rather than init to start the system. In runit's stage 2 process, runsvdir is started to manage everything under /service. All that is in the /service directory are symlinks to directories in /var/service which is where rpm packages place directories and information in order to run supervised services. These directories are named after the service they start. Unlike initscripts that can (and often do) start more than one daemon, each service starts one and only one service. So, for instance, when running LDAP you would use service ldap start on most Linux distributions, but on Annvix you would use srv --up slapd; srv --up slurpd. In this case there are two daemons, thus two services: slapd and slurpd.

Inside each directory is a single file called run. run is the equivalent of an initscript's start() function; all it does is start the service. The run script of spamd, for instance looks like this:

#!/bin/sh
PATH="/sbin:/usr/sbin:/bin:/usr/bin"

# this runs spamd supervised

OPTIONS="-c -m5 -H"

if [ -f ./env/OPTIONS ]; then
    OPTIONS=`head -1 ./env/OPTIONS`
fi

exec /usr/bin/spamd $OPTIONS 2>&1

Like an initscript, it sources certain configuration files to get options for the daemon (in this case, the /var/service/spamd/env/OPTIONS file). You also see that the daemon is started with exec and that STDERR is redirected to STDOUT. This is not always desirable, however, and needs to be dealt with on a case-by-case basis. In the case of spamd, a sub-service is also started that writes all of spamd's output to a logfile.

The logging service is another service that supervise handles, tied specifically to that particular service. This is done by using a log/ subdirectory of the service; so spamd's logging service run script is /service/spamd/log/run. The contents of this file are:

#!/bin/execlineb

# logging for the spamd service

/bin/foreground { /usr/bin/install -m 0700 -d -o logger -g logger /var/log/service/spamd }
/bin/cd /var/log/service
/sbin/chpst -u logger /sbin/svlogd -tt /var/log/service/spamd

This script basically takes STDOUT from spamd and writes it, through the svlogd tool (which handles the naming and automatic rotation of log files) to the directory /var/log/service/spamd. In this directory you will find a few files, some with odd names that are rotated logfiles, and one of interest, particularly current. The current logfile is the latest output of the service.

Keep this in mind if you need to debug a service or figure out why it isn't starting. Often errors from a service will be written to STDERR, which, if you remember, we are redirecting to STDOUT, which is then being written to the logfiles that svlogd is handling.

If you browse through the /var/log/service directory, you'll see a number of subdirectories. These subdirectories are log directories for other services, although not all services uses logging services (socklog is a notable exceptions). More often than not, the log files will be 0 bytes unless the program (like snort) is fairly verbose. As well, in many cases, services need to be started with debugging enabled in order to prevent them from forking into the background. In order for runsvdir to be able to control a service, it must not fork into the background.

For more detailed information on runsvdir, read the various manpages or visit the runit website.

<< Return to User Guide





Sponsors: Internet Marketing - Phoenix Landscaping - Phoenix Pools - Credit Cards