This documentation is outdated and has been moved to an updated version on https://linuxops.web.cern.ch/ .

Whatever you may find here is kept for historical purposes, but these docs are no longer relevant for many reasons after the service evolution and improvements .

Internal documents for Linux.Support

While this page should not contain any secrets, it is probably not very interesting to anybody outside of the Linux Support team ...

IT-OIS -> IT-CM

• https://twiki.cern.ch/twiki/bin/view/CMgroup/ItCmLcsSupportRota
• TasksRota2016

Meetings

• MeetingMinutes

Internal Procedures (access restricted to Linux Support members for most of these)

• DockerRegistry - configuration and management of docker/mirror-docker .cern.ch
• WhatToDocument - some ideas where we miss general user docs.
• ManageLinuxsoftBarn - running services description and common actions.
• ManageLinuxsoft - running services description, DNS alias mgmt.
• HowToUpdateWeb - making changes to http://cern.ch/linux pages.
• HowToTestDesktops - testing docs and results for all CERN stores desktops/laptops.
• [DEPRECATED] SoftwareUpdatesOnSLCOld - procedure on how to prepare and deploy updates.
• SoftwareUpdatesOnSLC - procedure on how to prepare and deploy updates.
• SoftwareUpdatesOnCC - procedure on how to prepare and deploy updates for CentOS Cern.
• SoftwareUpdatesOnSLCSpecialPackages - how to build packages for which we do not get SRMS from RedHat (java, acroread ...)
• SoftwareUpdatesOnRHEL - procedure on how to prepare and deploy updates.
• AllowAccessforRHE - how to give machines access to the licensed Red Hat Enterprise versions
• ChangeDefaultRelease - (NEEDS update) how to make a new subrelease the default
• LinuxSupportReferenceMachines - (NEEDS update) list of available reference machines (non-desktops)
• HowToTestPrintDrivers - sometimes the current "default" printer drivers produce blank pages. Here is how to test different ones.
• LinuxSupportKernelVersions - use git(1) for fast kernel comparisons
• Linux3rdLevel - privileged, fast-track Remedy flow for FIO service managers
• OvmRepositorySync - how linuxsoft sync OVM repository and how to add channel subscription online
• ConnectVirtualMachineConsole - How to use Microsoft Virtual Machine Manager
• ConfigureApacheSSLonSLC - Configure Apache with cern.ch/ca certificates SLC5 & SLC6
• PackagingCernlib - how to create the RPM from the AFS sources
• LxdistccNotesForServiceManagers - Service documentation for LXDISTCC
• LemonConfiguration - How to configure lemon on non-quattorized machine.
• UpdateMachineWeight - Change the machine weight for the CC.
• KojiMaintenance - Procedures to debug/fix common Koji problems.
• LinuxDistributionServer - lxdistXX
• LinuxDistributionServerWigner - lxdistwignerXX
• LinuxsoftNG - lxsoftXX
• ReleaseNewRedhatLinux - Procedure to add the new RHEL release to afs/aims2.
• BuildPackageAgainstSpecificKernel - Procedure to build SLC5/6 packaged against a specific kernel version.
• TechnicalNetworkTipsAndTricks - Jump Machine, etc...
• IscsiToolDebug - Cheatsheet for iscsi
• PxeHwreg - Live image boot for CF hwreg documentation.
• ContributePatchCernOpenafs - new procedure

Current Projects

• SLC57BuildObservations
• BuildingSystem
• SpecFileSRPMForKernelModule
• PulpEvaluation
• Syslinux6Evaluation

New in 2010

• Slc6Tests - Observations conerning the SLC6 test setup
• SpacewalkEvaluation - Taking a look on how systems management is done the Red Hat / Fedora way.
• MoreLinuxStatistics - Collecting and presenting detailed stats about our client systems
• JeanTheServant - Even better repository management - From sources via binaries via repositories to client system updates
• ExploringMSWindowsSynergies - How to get things working together with central CERN/IT MS product based services.
• CERNAlerterForLinux - Get these messages delivered on your desktop.
• TowardsDrupalPilotService

Ongoing

• Unify Linuxsoft and SWrep software repositories
• KernelModuleRPM : Procedure to create rpm for kernel module.

Older projects - would need some revisiting/updating perhaps ...

• DUPAddOns: RHEL5/SLC5 Driver Update Program Addons (new framework for addon kernel-modules).
• LinuxSmartCard: as triggered by IT-IS "pilot" and CNIC, look how/whether we can use Smartcards with Linux

Older (but still useful and used) Projects

• LinuxStatistics: collect usage info from various sources, generate shiny graphs.
• PrintingRevisitedAgain: do sthg about unfortunate (windows) CERN printing service...
• aims2: try to integrate AIMS better into the modern world: IT-CS DHCP server (done), CDB, frequent changes to images, (Dan)
• DistFsEval: network/distributed filesystems and clustering, (theoretical) evaluation
• LocalFsEval: local filesystems, access via RFIO (and some local test for Peter to show off)
• IncomingUpdateWorkflow: automate RPM incoming updates except for decisions, propagate to CDB/SWrep/Pakiti (Leszek, then Jarek)
• Lemonize reference machines and install servers (Jan)
• RootKitChecker: run checkrootkit (or something like it) on more machines: remove false positives, but stay with mainstream version. Integrate L.Cons tools.
• provide patch status overview to sec team (perhaps Pakiti can be used)
• IDSNetConnectionLogger: replace existing IDS (overloaded syscalls) (Alex)
• LinuxHardening: various things to prevent or limit (root-level) compromises
• FeatureListForSLC5: collecting point for new things that need to go into SLC5
• KerberosFuture: where should we go with the existing AFS KDC, user management etc
• UnifiedKerberos: look at what we can do against the two competing Kerberos realms at CERN
• LinuxTraining: draft proposal for a formal (HR) training curriculum
• GrubStage2Issues: the dreaded "grub loading stage 2" (and nothing later) error..
• improve NetDump: move to linuxsoft; do some automatic analysis of crashes, aggregate per crash type, feed back "XYZ crashed" to LEMON
• SysReqDiagnosis: how to interpret various SysReq-* things, such as collected by the operators on a stuck machine

• DistccPilotService: information about the distcc pilot service.
• PlayingWithPakiti2

Broken things and longstanding issues

(some overlap with the projects above (for things that actually work now), the things to add/fix for SLC4 and long-standing REMEDY tickets. All the broken things behind your OS you never wanted to know about...)

• NscdOnSlc5Core: the "nscd" daemon together with nss_ldap dies on SLC5, this is not good if we want to move to LDAP-based authorization
• CPUTimeBroken: sometimes the cpu time counters under /proc go bananas, and LSF then kills innocent jobs or rfiods
• SLC3LegacyServices: IT services that cannot or will not be on SLC4 in time for the EOL
• GssApiProblem: GSSAPI should replace Kerberos4 (and encapsulate Kerberos5), but DNS-aliased machines have trouble with ssh and cvs
• LinuxiSCSI: trying out linux iSCSI target/initiator code
• AFSCacheCorruption: Linux AFS clients seem to have a (rare) race which leads to corruption
• NetworkPerformance: R.Wartels performance drops on 2.6 (Red Hat #85033), A.Hirstius packet loss on access to /proc/net (Red Hat: wontfix for RHEL3)

• CompsXmlShortcomings: missing group requirements and/or badly placed packages make this almost unusable for anything else than the "Friss oder stirb" cern_recommended_setup.
• GlibcDnsLoadBalancing: getaddrinfo() sorts multiple A records according to longest matching prefix

Previous projects

• KerberosViaSSHNoAFS: end of June 2008 we see a marked increase of issues such as "no X11 forwarding after SSH" or "no AFS access after SSH" - believed to be fixed.
• instantclient RPMification (including tnsnames.ora)
• UcoRewrite: the venerable "uco" command needs updating/modularizing, still being used by user support

• NetDump: set up network crash dumps: requires new disk server, ncm-whatever for client-side config and some experience with the analysis tools.
• RH73Machines: make list of RH73 (and older) machines from various sources

Obsolete docs (kept here just 'in case')

This part of documentation is NOT UPDATED anymore, and most (it not all) of it is obsolete.

• Kerberos4PhaseOut: planning to get rid of Kerberos4 (we do not coordinate this activity anymore)
• HWTestTemplate0 (farm pcs), HWTestTemplate1 (desktop style), HWTestTemplateText, HWTestTemplateText1 (we do not test farm systems)
• SoftwareUpdatesOn7.3, (redhat 7.3 is gone since years)
• HowToRunASLC3MigrationCampaign - how to collect the hosts to conatct and how to send the warning mails (so is SLC3)
• RemoteAccessInLab: remote reset and serial console for the "lab" machines (these are gone)
• HowToTestFarmNodes (we do not test farm systems)
• AllowAccessforAIMS - how to add / remove users to AIMS (that was previous AIMS incarnation, not there anymore)
• FioLaDesktops - list of FIO-LA desktop/laptops (FIO-LA is no more)
• SpecBenchmark: how to run it on SLC3 (SLC3 is no more)
• ipw2200forSLC3: ipw2200 driver for SLC 3.0.x (ditto)
• SLC4MigrationCampaign: steps to get users out of SLC3 before we de-support it (as well)
• Quattorize (to some degree) installation servers (we wont quattor ourselves in the foot)
• ImprovementsForSLC4.X: what could be improved in or around SLC4X.
• FeatureListForSLC4: collecting point for new things that need to go into SLC4
• FunnyThingsOn64Bit: badness on IA64 that may/will affect EM64T/Opterons later
• CERNLIB needs to be a recompilable SRPM. Currently does not build npatchy, and a 64bit-version is an unknown horror (cernlib is now provided at CERN)
• FioLASLTaskList (one-off docs .. not needed anymore)

-- JaroslawPolok - 17-Feb-2010