Architecture

These servers provide only TFTP service for the moment.

The sync is provided by lsyncd (using inotify to rsync) on lxdist cluster.

Hardware

http://lxdistwigner02-ipmi.cern.ch/
http://lxdistwigner01-ipmi.cern.ch/

Installation

Kickstart

Note : The console parameter can be found on http://lemon.cern.ch web interface:

• search for your host
• click on the CDB template details
• click on include(serial_map_whatever)
• check parameters

Additional rpms

yum install mod_ssl apr apr-util apr-util-ldap http httpd-tools tftp-server tftp \
                 postgresql-docs postgresql-devel postgresql postgresql-server postgresql-libs \
                 perl-Frontier-RPC perl-XML-Writer createrepo mrepo \
                 koji-hub koji-hub-plugins koji-utils koji-plugin-sign koji koji-web \
                 mock mash koji-builder repoview \
                 rgmanager cman iscsi-initiator-utils iscsi-initiator-utils-devel mrepo \
                 shibboleth log4shib xmltooling-schemas opensaml-schemas mussh

Configure Lemon

* Add machine to foreman so it appears in Lemon interface. Execute from aiadm :

    ai-foreman-cli addhost $HOST --foreman-cookiejar ~/ibex/cookie.ai.judy.txt --operatingsystem "SLC 6.4" --arch x86_64 --hostgroup linuxsupport --owner toulevey --ptable "RedHat default"
    

• Verify pxe is disable (default should be disable but...)
• Check lemon web interface in the "Puppet" tab and then "Linuxupport":

Disable yum autoupdate

YUMHOUR=4
YUMUPDATE=0
YUMUPDATESECONLY=0
YUMONBOOT=0
YUMMAIL=1
YUMMAILTO="root"
YUMRANDOMWAIT=59
YUMCLEAN=1
YUMAPPLET=4

Configure tftp server

/etc/xinetd.d/tftp

Configure firewall

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth1 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 7001 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport http -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8443 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8488 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport ftp -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport rsync -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport tftp -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport syslog -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport sunrpc -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport sunrpc -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 32767 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 32767 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 32768 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 32768 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 32769 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 32769 -j ACCEPT
-A INPUT -m tcp -p tcp --dport 0:1023 -j DROP
-A INPUT -m udp -p udp --dport 0:1023 -j DROP
-A INPUT -m tcp -p tcp --dport 7100 -j DROP
-A INPUT -m udp -p udp --dport 7100 -j DROP
-A INPUT -m tcp -p tcp --dport 6000:6009 -j DROP
-A INPUT -m udp -p udp --dport 6000:6009 -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Disable selinux

Set to permissive:

/etc/selinux/config

Configure rsyslog

*.info;mail.none;authpriv.none;cron.none;local4.none;local5.none                /var/log/messages

#
$ActionQueueType LinkedList     # use asynchronous processing
$ActionResumeRetryCount 1      # infinite retries on insert failure
$ActionQueueSaveOnShutdown on   # save in-memory data if rsyslog shuts down
*.*     @@lxdist.cern.ch:5014
                                                                                                                  90,1          Bot

Change sysctl values

# Neighbour table overflow in logs.
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh1 = 1024

Troubleshooting

Synchronization from lxdist to lxdistwigner

• Login to lxdist and check where lsyncd runs

  [root@lxdist04 bin]# clustat
  Cluster Status for lxdist @ Thu May 30 15:16:12 2013
  Member Status: Quorate
  
   Member Name                                                     ID   Status
   ------ ----                                                     ---- ------
   lxdist01p                                                           1 Online, rgmanager
   lxdist02p                                                           2 Online, rgmanager
   lxdist03p                                                           3 Online, rgmanager
   lxdist04p                                                           4 Online, Local, rgmanager
   /dev/block/8:32                                                     0 Online, Quorum Disk
  
   Service Name                                             Owner (Last)                                             State         
   ------- ----                                             ----- ------                                             -----         
   service:aims2sync                                        lxdist01p                                                started       
   service:koji                                             lxdist03p                                                started       
   service:kojid01                                          lxdist01p                                                started       
   service:kojid02                                          lxdist02p                                                started       
   service:kojid03                                          lxdist03p                                                started       
   service:kojid04                                          lxdist04p                                                started       
   service:lsyncd                                           lxdist01p                                                started       
   service:lxdist                                           lxdist04p                                                started       
   service:lxdist01                                         lxdist01p                                                started       
   service:lxdist02                                         lxdist02p                                                started       
   service:lxdist03                                         lxdist03p                                                started       
   service:lxdist04                                         lxdist04p                                                started       
   service:pgsql                                            lxdist02p                                                started       
   service:tsmclient                                        lxdist01p                                                started       
     

• If not running restart it (clusvcadm -R ...)
• Check if lsyncd and aims2sync run on the same host (Owner column). If not restart them on the same node (clusvcadm -m ....)
• Check if /mnt/data2/etc/lsyncd.conf is correct:

     settings{
  	logfile = "/mnt/data2/log/lsyncd.log",
  	statusFile = "/mnt/data2/log/lsyncd-status.log",
  	statusInterval = 10 
  }
  
  sync{
  	default.rsyncssh, 
  	source="/mnt/data1/tftpboot", 
  	host="lxdistwigner01.cern.ch", 
  	targetdir="/mnt/data1/tftpboot",
  	rsync = {
  		verbose = true,
                  compress = false,	
  	}
  }
  
  sync{
  	default.rsyncssh, 
  	source="/mnt/data1/tftpboot", 
  	host="lxdistwigner02.cern.ch", 
  	targetdir="/mnt/data1/tftpboot",
  	rsync = {
                 verbose = true,
                 compress = false,
  	}
  }
  
     

Check if sync is ok

• Execute from lxdist (default is doing dryrun, so no risk) :

     /mnt/data2/bin/lxdist-sync-wigner
     

  It will rsync -n to the host and check if ok (please note the DIRECTORY are always newer because aims recreate them all the time ). If not ok check above

Change wigner alias

From lxdist machine:

/usr/sbin/dns-update
Usage: ./dns-update [--config ] --zone (internal|external|both) --alias  --iplist + ...
[root@lxdist04 sbin]# /usr/sbin/dns-update --zone internal --alias lxpxebootwigner --iplist 10.49.0.76

Reboot machines

• Update DNS:

/usr/sbin/dns-update --zone internal --alias lxpxebootwigner --iplist 10.49.0.76

• Wait 5 minutes and reboot 10.49.0.50
• when rebooted:

/usr/sbin/dns-update --zone internal --alias lxpxebootwigner --iplist 10.49.0.50

• Wait 5 minutes and reboot 10.49.0.76
• Last DNS update:

/usr/sbin/dns-update --zone internal --alias lxpxebootwigner --iplist 10.49.0.50 10.49.0.76