The instructions in this section were last reviewed on 2015-07-23. If you notice something is broken, or if you need help, please ask on the ``hn-cms-webInterfaces`` forum.
Request a VM from the CERN Openstack service
Go to openstack.cern.ch, request subscription if not yet done, and then sign in.
If you manage multiple projects, explicitly select “Current project: Personal <name>”.
Click on “Instances”, then on “+ Lauch Instance”. Fill up the form with:
After submitting the request, wait for the machine to build. Give it a few minutes and then check it is ready by looking at the console through openstack (click on “Instances”, then on the VM name and finally on “Console”).
Basic system install
SSH to your VM as yourself (your CERN account) and run the following commands to do basic system setup.
It requests a host certificate using your ~/.globus/user{cert,key}.pem, so make sure those files are up to date. While reading your private key, it asks for your password. If your certificate was issued by an authority other than CERN CA, map a non-CERN certificate to your account before attempting to run this step.
The Deploy command below will also check if it is possible to resize the current root partition on the fly, asking for your confirmation for doing so or skip. This is recommended because the default SLC6/5 installations do not size partitions to use the entire allocated disk. It is also useful if you later resize the VM to a different flavor (i.e. with more disk space). Note, however, the partition resizing procedure is not very reliable, so do never, ever, run it on a production server VM, nor expect it to work for VM images other than the SLC6/5 ones.
sudo -l
sudo yum -y install git.x86_64
mkdir -p /tmp/foo
cd /tmp/foo
git clone git://github.com/dmwm/deployment.git cfg
cfg/Deploy -t dummy -s post $PWD system/devvm
# OPTIONAL: review what happened: less /tmp/foo/.deploy/*
rm -fr /tmp/foo
# NOTE: CC7 don't has zip/unzip packages, if you need it apply
sudo yum install unzip.x86_64 zip.x86_64
If the above fails when running careq-cern-ch, then you can acquire host certificate manually. Here are the steps you need to do:
Generate private key/certificate with the following command:
openssl req -new -subj "/CN=`hostname -f`" -out newcsr.csr -nodes -sha512
Send the request to CERN CA: go to New Grid Host Certificate on https://gridca.cern.ch/gridca/ web site, click Request certificate using OpenSSL, then select host for which you request a certificate and send the form. On the next screen, Copy/Paste in the text box the content of file newcsr.csr generated by OpenSSL and click Submit. Finally, click Download Certificate Base 64 Encoded and save the file in the same folder as newcsr.csr. Then run the following set of commands:
sudo mv newcert.cer /etc/grid-security/hostcert.pem
sudo mv privkey.pem /etc/grid-security/hostkey.pem
sudo chmod 400 /etc/grid-security/hostkey.pem
sudo chown root:root /etc/grid-security/host{cert,key}.pem
rm newcsr.csr
Re-run the *Basic system install*.
Request proxy renewal rights for your VM
This step is optional.
If your server needs a proxy certificate to work, request the host to be included in myproxy.cern.ch. Write a message something like the one below, where HOSTNAME is the full host nameas shown by hostname -f. Run openssl x509 -in /etc/grid-security/hostcert.pem -noout -subject and verify the host certificate is correct in the message body! The request usually takes one to several days to fulfill. In the mean time you can continue to use the machine, but you have to renew the proxy manually as per step 8.
From: Your Name <you@cern.ch>
To: px.support@cern.ch
Cc: cms-service-webtools@cern.ch
Subject: myproxy registration request for HOSTNAME
Could you please add the following host certificate to myproxy.cern.ch
trusted_retrievers, authorized_retrievers, authorized_renewers policy?
This is a development server for CMS web services and requires use of
grid proxy certificates.
/DC=ch/DC=cern/OU=computers/CN=HOSTNAME.cern.ch
Regards,
Your Name
Log out
Log out. It is also recommended to reboot the server, specially if you have resized partitions on the fly, but it isn’t really necessary.
If you use SSH persistent connections, be sure to terminate the master connection before logging in again. The commands in step 2 modify users and groups, and it’s important they apply when you execute the next steps. This requires completely new log-in, and using persistent SSH connections may cause you to use cached credentials.
Log in
SSH into your VM again. Run the following to double check you got the multiple service accounts.:
id # should print out large number local _foo groups now
Set up authentication
This step is optional.
If you develop a server with secrets, grab them now. You only need the secrets for the servers you manage, such as /data/auth/t0wmadatasvc/t0auth.py. However you do not need this at all - you can just proceed to next step, which will then create dummy auth info that you can then overwrite with real data. If you do create the auth directory, it must be adequately protected and _sw group readable, so do run all the chmod/chgrp commands shown below.
mkdir -p /data/auth
mkdir -p /data/auth/wmcore
mkdir -p /data/auth/myservice
vi /data/auth/myservice/mysecret # e.g. /data/auth/t0wmadatasvc/t0auth.py
# or grab them all:
# rsync -avu cmsweb@vocmsNNN.cern.ch:private/conf/ /data/auth/
chgrp -R _sw /data/auth
chmod ug=r,o-rwx $(find /data/auth -type f)
chmod u=rwx,g=rx,o-rwx $(find /data/auth -type d)
Note that you normally create the info by hand, instead of copying from the cmsweb account as only the admins can do the latter. The important thing is you set up directory structure /data/auth with the secrets info that you need, for only those services you plan to install. Also note that you need to create a blank /data/auth/wmcore directory if you are deploying any wmcore based service or deploying the frontend. This directory holds the key used by both frontends and backend services to hmac verify the http headers. A blank directory there tells the deploy to generate a new key.
Software installation
The following installs standard multi-account setup using the deployment/configuration from the HG1509a release, and the corresponding RPMs from the comp.pre repository. You may pick up any release tag you need. The list of comp release tags can be found here.
Note you will be asked for the privkey passphrase in case your service requires a proxy certificate to work.
If you set up authentication in step 6, add the ``-a $PWD/auth`` option in the InstallDev command below.
# Get the configuration
(cd /data; git clone git://github.com/dmwm/deployment.git cfg && cd cfg && git reset --hard HG1509a)
# Deploy services
(VER=HG1509a REPO="comp.pre" A=/data/cfg/admin;
cd /data;
$A/InstallDev -R comp@$VER -s image -v $VER -r comp=$REPO -p "admin frontend das mongodb")
# start / status / stop
(A=/data/cfg/admin; cd /data; $A/InstallDev -s start)
To install the full set of services use the -p argument with: admin frontend couchdb confdb bigcouch das dbs dqmgui mongodb phedex sitedb reqmgr workqueue reqmon alertscollector crabserver crabcache dmwmmon asyncstageout t0wmadatasvc dbsmigration t0_reqmon acdcserver reqmgr2 gitweb
Note that as per 2015-July the default (production) architecture is slc6_amd64_gcc481. If you need to use a different one, you must specify it with -A and point to the correct repository correctly with $REPO. For instance, to install the gcc493 arch, switch is ships python2.7, you’d use:
(VER=HG1509a REPO="comp" A=/data/cfg/admin;
cd /data;
$A/InstallDev -R comp@$VER -s image -v $VER -r comp=$REPO -p "admin frontend das mongodb")
Proxy renewal
This step is optional.
On deployments of services that require a proxy certificate to work, the procedure in the previous step uploads the user credentials to myproxy that are valid for one month (aka long term proxy). The VM, in turn, will keep renewing a short term proxy until the long term one expires.
You will get mail notifications days before it expires, though. Upon reception of such messages, run the procedure below to renew the long term proxy. If your request in step 3 has not been handled yet, you’ll need to run it every 36 hours until the machine gets included into myproxy.cern.ch. You can also run it at any time even when the proxy is not yet about to expire.
cd /data
$PWD/cfg/admin/ProxySeed -t dev -d $PWD/srv/current/auth/proxy
Manage servers
Check server status, start servers using these commands:
cd /data
$PWD/cfg/admin/InstallDev -s status
$PWD/cfg/admin/InstallDev -s start
Clean up
To clean up state, create VM snapshots and roll back to suitable point in time, scrap the VM and recreate it, or use the following commands to roll things back to where they were until step 7. WARNING: The rm command will wipe out almost everything on /data - MAKE SURE you run it in right place, and want to run it!
([ "$(hostname -s)" = "mydevvmname" ] || exit; echo "Deleting..."; cd /data; $PWD/cfg/admin/InstallDev -s stop; crontab -r; killall python; sudo rm -rf srv)
Repeat steps 7 to 10 for any new software versions. You can use private RPM repository such as comp.pre.yourlogin to exercise builds which haven’t been synced back to comp.pre yet. See Developing Against RPMS for details on how to upload to private repositories.
These instructions may be outdated since we don’t support them. Please contribute back your corrections, in case of any.
These instructions create Scientific Linux 5.7 virtual machine under VirtualBox 4.1.x. You can another hypervisor if you prefer; the translation should be very straightforward. VirtualBox is easy to use and free for uses such as this, so an attractive choice if you don’t have another hypervisor. The instructions are somewhat geared towards CERN-like environment. You can adjust them to your local site conventions as far as groups, time servers, and such.
First create a new virtual machine labelled SL5.7, Linux/RedHat (64-bit), at least 2048 MB RAM. Create a new start-up disk: VDI, dynamically allocated, 40 GB in size. Download the install boot image and save it as boot_sl7_x86_64.iso in your downloads folder. Attach it on IDE into your VM. Set networking to bridged mode, and give the VM a pre-allocated fixed MAC address.
There are three main reasons for bridged networking and a preallocated MAC address. The first is that it’s a great deal easier to SSH into and use the web server in the VM when your VM appears as any other server on your LAN with bridged networking. The second reason is that in order to get a host certificate for your VM, your site will likely require you to register the hostname and the MAC address. Specifically many sites, including CERN, will not grant a host certificate for a laptop. Third is that in order to get myproxy renewal rights, your host needs a stable name, and to get one you typically need a pre-registered MAC address.
Install minimal SL5.7 server into the VM using the boot image:
- Language: English; Keyboard: us; Method: HTTP, DHCP no IPv6, linuxsoft.cern.ch, /cern/slc5X/x86_64/
- OK to initialise partition table
- Remove all partitions and create default layout: sda1:/boot 101 MB, sda2:LVM VG00 [LV01 swap 4000 MB, LV00 / ext3 rest]
- Install grub loader [default]
- Network: eth0, IPv4 DHCP, IPv6 Disabled, hostname via DHCP [default]
- Region: Europe/Zurich, system clock uses UTC [default]
- Set root password
- Installation: server, customize now
- Clear everything in: Desktop environments, Servers, Cluster Storage, Clustering, SLC Customizations
- Applications: Text-based Internet (only)
- Development: Development Libraries, Tools (only)
- Base System: Administrative Tools, Base, Java (only)
- After install remove CD, reboot into first boot:
- Authentication: MD5 + shadow (no kerberos);
- Firewall: enabled, SELinux: enforcing; Customize: ssh, Other ports: empty (remove afs3-callback:udp)
- Keyboard: U.S. English
- Network: DNS: Hostname: (give a name) (all other defaults)
- System services: (defaults) + ntpd
- Timezone: Europe/Zurich, system clock uses UTC; use ntp, servers: ip-time-{0,1,2}.cern.ch
- Sound card: defaults (Intel 82801AA-ICH)
Now login as root and run the following, possibly adjusted for your site:
vi /etc/ntp.conf # server ip-time-{0,1,2}.cern.ch
service ntpd restart
yum -y update
yum -y install zsh
yum -y clean packages
vi /etc/sudoers # uncomment "%wheel ALL=(ALL) NOPASSWD: ALL"
ME=<your_afs_login>
echo your.account@cern.ch > /root/.forward
groupadd -g 1399 zh
useradd -M -g zh -G wheel -s /bin/zsh -u 12345 -c "Your Name" -d /home/$ME $ME
passwd $ME
mkdir -p /home/$ME /data
chown -R $ME:zh /home/$ME /data
# install guest additions
mount /dev/cdrom /media && cd /media
sh ./VBoxLinuxAdditions-amd64.run
cd /; umount /dev/cdrom
# upgrade zsh (optional)
cp -p /bin/zsh{,.old}
cd /tmp
wget http://downloads.sourceforge.net/zsh/zsh-4.3.12.tar.bz2
tar jxf zsh-*.tar.bz2
cd zsh-*/
./configure --prefix=/usr --libdir=/usr/lib64 zsh_cv_sys_link=no
make -j 2
make install # DESTDIR=/tmp/foobar for test
rm -f /bin/zsh; ln /usr/bin/zsh /bin/zsh
rm -fr /tmp/zsh*
# turn off
shutdown -h 0
Create VM snapshot for installed state. Restart. Run post-install, e.g. copy your shell environment:
scp ~/.z{log{in,out},sh{env,rc}} your-vm-host:
scp -rp ~/.globus your-vm-host:
Your VM is ready for use. SSH into it and deploy servers normally as per dev-vm instructions:
# one-time preparation
mkdir -p /tmp/foo
cd /tmp/foo
svn co svn+ssh://svn.cern.ch/reps/CMSDMWM/Infrastructure/trunk/Deployment cfg
sudo -l
cfg/Deploy -t dummy -s post $PWD system/devvm
rm -fr /tmp/foo
sudo yum -y install voms-clients myproxy
B=/afs/cern.ch/project/gd/LCG-share/3.2.8-0
sudo scp -rp you@lxplus.cern.ch:$B/glite/etc/vomses /etc/vomses
sudo scp -rp you@lxplus.cern.ch:$B/external/etc/grid-security/vomsdir /etc/grid-security
# server installation, using admin tools as shortcuts
cd /data
rsync -avu cmsweb@lxplus.cern.ch:private/conf/ /data/auth/
chgrp -R _sw /data/auth
chmod ug=r,o-rwx $(find /data/auth -type f)
chmod u=rwx,g=rx,o-rwx $(find /data/auth -type d)
A=/data/cfg/admin REPO="-r comp=comp.pre" VER=1111d
PKGS="admin frontend base couchdb das dbs dbsweb dqmgui filemover mongodb phedex"
PKGS="$PKGS overview sitedb/legacy stagemanager t0datasvc t0mon reqmgr workqueue"
$A/InstallDev -s image -v hg$VER -a $PWD/auth ${=REPO} -p "$PKGS"
$A/InstallDev -s start
$A/InstallDev -s status
# cleanup
cd /data
$A/InstallDev -s stop
crontab -r
killall python
sudo rm -fr [^aceu]* .??* current enabled
These instructions may be outdated since we don’t support them. Please contribute back your corrections, in case of any.
This is really not a virtual machine environment, but there is experimental support for settings this up on an OS X laptop. This has only been tested with Snow Leopard:
# Fake enough of grid environment
sudo mkdir -p /etc/grid-security
B=/afs/cern.ch/project/gd/LCG-share/3.2.8-0
GS=/etc/grid-security BGS=$B/external/etc/grid-security
sudo rsync -av --delete you@lxplus.cern.ch:$B/../certificates $GS/certificates/
sudo rsync -av --delete you@lxplus.cern.ch:$B/glite/etc/vomses/ /etc/vomses/
sudo rsync -av --delete you@lxplus.cern.ch:$B/glite/etc/vomses/ /etc/vomses/
sudo rsync -av --delete you@lxplus.cern.ch:$BGS/vomsdir/ $GS/vomsdir/
sudo chown -R root:$(id -gn root) /etc/grid-security /etc/vomses
# Create accounts and all the rest; this installs into /users/cmssw/test
# instead of using /data. You may need to iterate and copy a host cert
# from somewhere into machine if the default rule doesn't work.
mkdir -p /tmp/foo
cd /tmp/foo
svn co svn+ssh://svn.cern.ch/reps/CMSDMWM/Infrastructure/trunk/Deployment cfg
sudo -l
CMS_DEV_ROOT=/users/cmssw/test cfg/Deploy -t dummy -s post $PWD system/devmac
cd; rm -fr /tmp/foo
# Install software using roughly standard dev-vm instructions.
cd /users/cmssw/test
rsync -avu cmsweb@lxplus.cern.ch:private/conf/ $PWD/auth/
chgrp -R _sw $PWD/auth
chmod ug=r,o-rwx $(find $PWD/auth -type f)
chmod u=rwx,g=rx,o-rwx $(find $PWD/auth -type d)
cd /users/cmssw/test
A=$PWD/cfg/admin REPO="-r comp=comp.pre" VER=1111a
PKGS="admin frontend base couchdb das dbs dbsweb dqmgui filemover mongodb phedex"
PKGS="$PKGS overview sitedb/legacy stagemanager t0datasvc t0mon reqmgr workqueue"
$A/InstallDev -s image -v hg$VER -a $PWD/auth ${=REPO} -p "$PKGS"
$A/InstallDev -s start