Installation of a CCPC server for SLC4

Preliminaries

The LHCb Creditcard PCs are embedded disk-less x486 compatible PCs. They boot via the network and they use an NFS tree installed on a remote server as their root filesystem.

The system we are using is based on the Redhat package system-config-netboot. It is however not required that your server is running Redhat. The Creditcard PCs will run Scientific Linux CERN 4, centrally and automatically up to date from CERN.

Hardware setup

I would like to remind you here of the Ethernet hardware problem. This means that you will not be able to run the CCPC on long (> 10 m) cables on 100 MBit Ethernet. If you are not sure how long the cable between CCPC and the next active element (switch, router, PC) is, then it is recommended to add a hub or switch in front of the CCPC. Please keep in mind, that if you have structured cabling in your office/lab, where you plug the cable in a wall-plug, there can be up to 90(!) m of cable until the next patch-panel.

Symptoms for Ethernet problems

You can tell that you are hit by the Ethernet problem, when you see either of the following In all these cases the problem can be cured by adding an active element close to the CCPC.

Before you start

In the following examples it is assumed that you have an rpm based system. If this is not the case, most things should be easy to adapt. In case of problems do not hesitate to contact me.

You will need root privileges on the server. The server must be able to connect to the CERN web-servers.

Check that the following packages are available on your system: nfs-utils, tftp-server, dhcp and syslog. On my machine a check gives 

[pclhcb74] ~ % rpm -q tftp-server dhcp nfs-utils sysklogd
tftp-server-0.39-1
dhcp-3.0.1-38_EL4
nfs-utils-1.0.6-65.EL4
sysklogd-1.4.1-26_EL
Your version numbers may differ, this is fine. If any of these packages are missing you need to install them. You can find them on http://rpmfind.net.

Make sure that you have enough diskspace. You will need 2 Gigabytes for the basesystem and ~ 10 to 50 MB per Creditcard PC. The server PC need neither be fast nor have a lot of memory. If you have only a few CCPCs to serve almost any Linux machine will do.

Installing the CCPC server software

Create a directory for the install and define an environment variable. 
mkdir /opt/ccpc
export CCPCBASE=/opt/ccpc
mkdir $CCPCBASE/snapshot
Make sure that /sbin and /usr/sbin are in your path. This is assumed throughout this page! 
export PATH=/usr/sbin:/sbin:$PATH

Download the software

Download the following files to a temporary area, but do not unpack them yet.

slc4-ccpc-root.tar.bz2
slc4-2.6.9-1.EL.ccpc.nbi
MD5SUMS

Use md5sum to verify the integrity of your downloads. 

md5sum -c MD5SUMS

Install the boot image and root-filesystem

You must install the boot image to the place where tftpd can pick it up (typically /tftpboot). 
cp slc4-2.6.9-1.EL.ccpc.nbi /tftpboot/slc4-2.6.9-1.EL.ccpc.nbi
The root filesytem must be placed into the base directory CCPCBASE 
cd $CCPCBASE
tar -jx < /tmp/slc4-ccpc-root.tar.bz2
export CCPCROOT=$CCPCBASE/root

The CCPCROOT and CCPCBASE environment variables should be added to the environment of the account, used for administering the CCPCs. In the following it will be assumed that they are defined.

Configure the root filesystem

The root filesystem must be slightly adapated to fit your server. This is done using a little script: fixupsrv. It will create a local user called cc on your CCPCs and will ask your for a password for this user. 
sudo ./fixupsrv $CCPCROOT
Which password do you want to change?

        a - AFS [default]
        u - local UNIX

Enter the appropriate letter or hit Return for the default: u
Changing password for user cc.
New UNIX password: 
Retype new UNIX password:
It is probably a good idea to create an identical user on the server and share its home directory with the CCPCs. In this way you can easily interchange files with the CCPCs. A description can be found here.

Configuring the server

NFS

The /etc/exports file must as a minimum contain the following two lines

/opt/ccpc/root *(no_subtree_check,ro,sync,no_root_squash,insecure)
/opt/ccpc/snapshot *(rw,sync,no_root_squash)

Details of the options vary with the version of NFS used. The keypoint is that snapshot must be writeable and root read-only. Exporting to everybody *(rw) is not the best of all ideas, unless you are in a reasonably secure, fire-walled environment.

DHCP

The dhcp server will provide all information about the network to the CCPCs. I recommend to define a group for the CCPCs in your /etc/dhcpd.conf.

Unfortunately the exact configuration of the dhcp server depends strongly on your network setup. Here an example dhcpd.conf which works at CERN, your mileage may vary. One essential point is the correct root-path option in dhcp. It must point to $CCPCBASE and NOT to $CCPCROOT. The value of $CCPCBASE used in this page would require the following line in /etc/dhcpd.conf 

option "137.138.142.66:/opt/ccpc";

The dhcp, time, portmap, tftp and nfs servers must be started respectively restarted. On RedHat systems you can use the service command. 

sudo /sbin/service portmap restart
sudo /sbin/service nfs restart
sudo /sbin/service dhcpd restart
sudo /sbin/chkconfig time on   # RFC 868 compliant time server internal to xinetd
sudo /sbin/service xinetd restart
sudo /sbin/chkconfig tftp on
You might want to configure your server such that the nfs and dhcp services are started automatically at boot-time. On Redhat systems this is done by: 
sudo /sbin/chkconfig nfs on
sudo /sbin/chkconfig dhcpd on

Odds and ends

Log messages from the CCPCs will sent to the syslog daemon of your server. For this to work it must be started with the -r option. On Redhat systems you have to add this to SYSLOGD_OPTIONS in /etc/sysconfig/syslog. After this change you have to restart the syslog server: 

sudo killall -s HUP syslogd

Firewalls and other horrors

The default settings for firewalls nowadays are quite restrictive. Unfortunately the standard scripts like chkconfig will not open the ports, should they happen to be blocked. So it is possible that nothing will work. Try (momentarily) to switch off the firewall on the server. 
/sbin/service iptables stop
If that cures it, open the ports required for the working of nfs, dhcpd, time and nfs and restart the firewall.

Installing the Creditcard PC software

 Finally the Creditcard PC base software must be installed. All standard software is distributed in the same way as the system software, that is as rpm packages using yum. In the following it is assumed that the CCPCROOT environment variable points to the root of the CCPC software tree.

Make sure that the root-filesystem of the CCPCs has the same nameserver config as the one 

sudo chroot $CCPCROOT bash -c "yum -y install ccpc"
That's it! Yum is a nifty tool, which knows how to resolve dependencies.

To keep the server up-to-date you can add the above line to the crontab of a user with sudo priviledges (once a day for example).

Additional software

The software distribution is pretty complete. Should you however miss something than you can simply install it using yum. As an example we will install the (oddly) popular nedit editor. 
sudo chroot $CCPCROOT bash -c "yum -y install nedit"
This will trigger the following output: 
Gathering header information file(s) from server(s)
Server: SLC4 CCPC packages
Server: SLC4 extras packages
Server: SLC4 os packages
Server: SLC4 updates packages
Finding updated packages
Downloading needed headers
Resolving dependencies
Dependencies resolved
I will do the following:
[install: nedit 5.4-3.i386]
Is this ok [y/N]: y
Downloading Packages
Getting nedit-5.4-3.i386.rpm
nedit-5.4-3.i386.rpm      100% |=========================| 509 kB    00:00     
Running test transaction:
Test transaction complete, Success!
nedit 100 % done 1/1 
Installed:  nedit 5.4-3.i386
Transaction(s) Complete

Adding Creditcard PCs to the server

 Just get and assign a fixed IP address to each CCPC in your system. Attention: truly dynamic addresses will most likely not work with NFS.

  1. Add the CCPC with its hardware (a.k.a Ethernet) address IP name and IP address to the dhcpd.conf file. Check the example for pclbcc00.
  2. Restart the dhcpd server 
    /sbin/service dhcpd restart
  3. Boot the CCPC

Sharing a user home directory between server and CCPCs

 In this section we assume that you want to create the CCPC default user cc on your server. Its homedirectory is to be shared r/w with the CCPCs. First create with the same uid/gid as on the CCPC. If this conflicts with your local settings, you need to change it also on the CCPCs (using chroot). 
sudo groupadd -g 20502 cc
sudo useradd -u 60001 -g cc cc
This creates a home-directory in /home/cc. Add the following line to the server's /etc/exports file. 
/home/cc   *(rw,sync)
You might want to tighten the security for this share. Next export the directory: 
sudo /usr/sbin/exportfs -ra
Finally mount the /home/cc directory on the server by declaring it in the $CCPCROOT/etc/fstab file of the CCPCs. For example type something like this on the server: 
sudo bash -c "echo $(hostname):/home/cc /home/cc nfs
rw,rsize=8192,wsize=8192 0 0 >> $CCPCROOT/etc/fstab"
After rebooting the CCPC, or, more civilized, doing a sudo mount -a as cc on the CCPC, you should be able to read and write to /home/cc from both the server and the CCPC.

Credits

Many thanks to Will Bell (Glasgow) for a lot of corrections and useful suggestions!

This page last updated 15-12-2005
Comments & questions to Niko Neufeld