Koji

Koji integration

Linuxsoft

Egroups

User management

Web interface

Connect with firefox : https://koji.cern.ch/koji/

Authentication is done with Kerberos so you need to add in "about:config" two options:

network.negotiate-auth.delegation-uris = cern.ch
network.negotiate-auth.trusted-uris = cern.ch

Add user account

STEP 1 : koji add-user --principal=<username>@CERN.CH <username> 

Please note the user has been created automatically if he tried to login in the web interface so you may get:

 GenericError: user already exists: user

You can safely execute the second command to give them the required permission:

STEP 2 : koji grant-permission build <username>

Redirect user to the documentation: https://twiki.cern.ch/twiki/bin/view/LinuxSupport/BuildingRPMswithKoji And warn them that it is a BETA service.

Add admin account

koji add-user --principal=<username>@CERN.CH <username> 
koji grant-permission admin <username>

Create new tag

# vi /mnt/data2/home/build/bin/tags/createtag5.sh
# vi /mnt/data2/home/build/bin/tags/createtag6.sh
# vi /mnt/data2/home/build/bin/tags/createtag7-el7.sh

Machines

koji hub: koji.cern.ch
builder: lxdist01 lxdist02 lxdist03 lxdist04

fstab

/dev/mapper/vg_lxsoft06-root /                       ext4    defaults        1 1
UUID=fa79925c-fcbe-4b8d-8c37-30c0d25f2e2a /boot                   ext4    defaults        1 2
/dev/mapper/vg_lxsoft06-tmp /tmp                    ext4    defaults        1 2
/dev/mapper/vg_lxsoft06-var /var                    ext4    defaults        1 2
/dev/mapper/vg_lxsoft06-swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
UUID=75c859da-fd35-4d96-9770-06aa1e28ef60 /mnt/koji     ext4    defaults        1 1
/dev/mapper/mirror-lvol0  /mirror                       ext4    defaults        1 1

• Verify if mirror-lvol0 is mounted in /mirror
• Verify that /mnt/koji is mounted
• Verify /etc/exports

       /mnt/koji      *.cern.ch(rw,all_squash)
       

apache

• Verify if apache runs correctly koji wsgi. http://lxsoft06.cern.ch/koji/ and http://lxsoft06.cern.ch/kojifiles/
• Verify that it serves /mirror/mash on http://lxsoft06.cern.ch/mash/
• Verify that it serves /mirror/buildsys on https://lxsoft06.cern.ch/buildsys/

kojira

It regenerate automically koji internal repository when packages are added

• ps -eaf | grep kojira
• check /var/log/kojira.log for errors.

mash

It generate repository for our tags and is used by Hamster/ Agile team

• /root/bin/execute-mash.sh
• executed every 5 minutes

In case of failure

• stop cron jobs
• killall mash instances
• rm -rf /mirror/mash/tag

       tag = ai5  ai6  aifc16  aifc17  el5-experimental  el6-experimental  hw5  hw6  lcgdm5  lcgdm6  lemon5  lemon6  lemonfc16  lemonfc17
      

• cd /mirror/mash/ and execute

  mash tag

builder kojid

• check if kojid runs
• check logs in /var/log//kojid.log

Signing key management

• backup /mnt/data2/etc/koji-hub/gnupg just in case
• execute:

      export GNUPGHOME=/mnt/data2/etc/koji-hub/gnupg
      gpg --import public.asc
      gpg --import private.asc
      

• Test that you can sign manually:

      rpm --resign --define '_signature gpg' --define '_gpg_name CERN MIG ' --define '_gpgbin /usr/bin/gpg' --define '_gpg_path /mnt/data2/etc/koji-hub/gnupg/'  mytest.rpm 
      rpm -K mytest.rpm 
     

• Verify permision on $GNUPGHOME

     # ls -ld $GNUPGHOME
     drwx------. 3 apache apache 4096 Feb 26 14:59 gnupg
     # ls -l $GNUPGHOME
     rw-------. 1 apache apache 7856 Oct 26  2012 gpg.conf 
     drw-------. 2 apache apache 4096 Oct 26  2012 private-keys-v1.d
     -rw-------. 1 apache apache 2342 Feb 26 10:11 pubring.gpg
     -rw-------. 1 apache apache 2342 Feb 26 10:11 pubring.gpg~
     -rw-------. 1 apache apache  600 Oct 26  2012 random_seed
     -rw-------. 1 apache apache  968 Oct 26  2012 RPM-GPG-KEY-agileinf
     -rw-------. 1 apache apache 2406 Feb 26 10:03 RPM-GPG-KEY-mig
     -rw-------. 1 apache apache 3880 Feb 26 10:17 secring.gpg
     -rw-------. 1 apache apache 1280 Feb 26 10:11 trustdb.gpg
  
     

• Define the tags you need to sign in /mnt/data2/etc/koji-hub/plugins/sign.conf like this:

[ai6-testing]
rpm = /bin/rpm
gpgbin = /usr/bin/gpg
gpg_path = /mnt/data2/etc/koji-hub/gnupg
gpg_name = CERN agileinf 
gpg_pass = ****************
enabled = 1

Common problems

Q: My kernel module is not building anymore, and ask for latest kernel version.

• Check koji builder and upgrade them to latest kernel. using wass on lxadm.
• Check spec file is correct

Q: Tasks stay in purple in web interface and in "Free" state

• Restart kojid on builder /etc/init.d/kojid restart
• Check free space on /build (puppet should take care of that through cron.)

Q: Package X-Y-Z is missing but it seems it should be available in the repo.

• Check if it is tagged properly. Some package are built for tag el6_3 and are not then tagged on el6 e.g : koji tag-pkg el6 expat-2.0.1-11.el6_2 It should be fixed for all package built from mid-july 2012.
• Koji is using a slc6 mirror from linuxsoft.cern.ch in lxsoft06:/mirror/
• Check the package has not been blocked for the specific tag e.g : koji list-pkgs --show-blocked --tag=ai6

Q: This error message appears : GenericError: hash changed for external rpm: autoconf-2.59-12.noarch@el5-external-repo-os-2 (00f623da00db01162455899eecd744e2 -> de86fbcf57041ef7e0695783699fce1a)

• It means RPM in our external repo (linuxsoft) have changed. The easiest way is to delete the corresponding el5-external-repo-os-2 and re-add the repo to the corresponding tag.

       koji remove-external-repo el5-external-repo-os-2
       koji add-external-repo -t el5-build el5-external-repo-os-3 "http://koji.cern.ch/cern/slc5X/\$arch/yum/os/"
       

Q: How to push updates from Koji to Linuxsoft? Add a Koji administrator to run:

• First, run ssh root@lxdist01 su - build /mnt/data2/koji/bin/execute-mash.sh
• Run ssh root@lxsoft04 linuxsoft-sync-rhn -s koji

Q: I have strange auth error in the logs

Nov 19 09:32:50 lxdist01 httpd[8845]: [error] 2013-11-19 09:32:50,874 [WARNING] m=listTasks u=None p=8845 r=137.138.144.42:60050 koji.xmlrpc: Traceback (most recent call last):
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]   File "/usr/share/koji-hub/kojixmlrpc.py", line 212, in _wrap_handler
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]     response = handler(environ)
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]   File "/usr/share/koji-hub/kojixmlrpc.py", line 255, in handle_rpc
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]     return self._dispatch(method, params)
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]   File "/usr/share/koji-hub/kojixmlrpc.py", line 279, in _dispatch
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]     self.check_session()
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]   File "/usr/share/koji-hub/kojixmlrpc.py", line 261, in check_session
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]     context.session = koji.auth.Session()
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]   File "/usr/lib/python2.6/site-packages/koji/auth.py", line 112, in __init__
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]     raise koji.AuthError, 'Invalid session or bad credentials'
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] AuthError: Invalid session or bad credentials
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]

• Verify that "kojira" run on the correct node in the cluster.
• clustat | grep koji
• kill kojira on other nodes.

Q: I strange python error in the root.log "Not using downloaded repomd.xml because it is older than what we have"

Executing command: /usr/bin/repoquery -c /var/build/mock/el6rt-build-44358-31748/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/build/mock/el6rt-build-44358-31748/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
  Not using downloaded repomd.xml because it is older than what we have:
DEBUG util.py:264:    Current   : Tue Apr 29 10:41:52 2014
DEBUG util.py:264:    Downloaded: Thu Apr 24 14:54:54 2014
DEBUG util.py:264:  Traceback (most recent call last):
DEBUG util.py:264:    File "/usr/bin/repoquery", line 1241, in 
DEBUG util.py:264:      main(sys.argv)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 1235, in main
DEBUG util.py:264:      repoq.runQuery(regexs)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 805, in runQuery
DEBUG util.py:264:      pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 742, in matchPkgs
DEBUG util.py:264:      pkgs = self.returnPkgList(patterns=items)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 695, in returnPkgList
DEBUG util.py:264:      pkgs = self.pkgSack.returnNewestByNameArch(**kwargs)
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 887, in 
DEBUG util.py:264:      pkgSack = property(fget=lambda self: self._getSacks(),
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 669, in _getSacks
DEBUG util.py:264:      self.repos.populateSack(which=repos)
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/repos.py", line 308, in populateSack
DEBUG util.py:264:      sack.populate(repo, mdtype, callback, cacheonly)
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 165, in populate
DEBUG util.py:264:      if self._check_db_version(repo, mydbtype):
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 223, in _check_db_version
DEBUG util.py:264:      return repo._check_db_version(mdtype)
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1256, in _check_db_version
DEBUG util.py:264:      repoXML = self.repoXML
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1455, in 
DEBUG util.py:264:      repoXML = property(fget=lambda self: self._getRepoXML(),
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1447, in _getRepoXML
DEBUG util.py:264:      self._loadRepoXML(text=self)
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1437, in _loadRepoXML
DEBUG util.py:264:      return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1412, in _groupLoadRepoXML
DEBUG util.py:264:      if self._commonLoadRepoXML(text):
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1250, in _commonLoadRepoXML
DEBUG util.py:264:      self._revertOldRepoXML()
DEBUG util.py:264:    File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1083, in _revertOldRepoXML
DEBUG util.py:264:      os.rename(old_data['old_local'], old_data['local'])
DEBUG util.py:264:  OSError: [Errno 2] No such file or directory

• run koji regen-repo XYZ-build, with XYZ being the repo in question (can be found in the first line of the log, for this example el6rt)

SSO configuration

0/ Verify you have a valid SSL certificate
   https://gridca.cern.ch/gridca/

1/ Register your application
   http://www.cern.ch/sso-management
   Register New SSO Application
    Application name: https://koji.cern.ch
    Application URI: https://koji.cern.ch/mellon
    Service provider: SAML2
    Application homepage: https://koji.cern.ch
    Application description: koji

2/ Get Saml certificate:
   cd mnt/data2/etc/httpd.koji/conf.d/mellon
   /usr/libexec/mod_auth_mellon/mellon_create_metadata.sh https://koji.cern.ch/mellon https://koji.cern.ch/mellon

you will get three files:

https_koji.cern.ch_mellon.cert
https_koji.cern.ch_mellon.key
https_koji.cern.ch_mellon.xml

3/ Get Metadata
   wget https://login.cern.ch/FederationMetadata/2007-06/FederationMetadata.xml
   (or wget https://login-dev.cern.ch/FederationMetadata/2007-06/FederationMetadata.xml)

4/ Serve the xml file on https://koji.cern.ch/metadata

mkdir /var/www/html/metadata
cp https_koji.cern.ch_mellon.xml  /var/www/html/metadata/koji.xml

Verify that https://koji.cern.ch/metadata/koji.xml is correctly served.

5/ Configure Apache


        MellonSetEnvNoPrefix "ADFS_GROUP" "http://schemas.xmlsoap.org/claims/Group"
        MellonSetEnvNoPrefix "ADFS_EMAIL" "http://schemas.xmlsoap.org/claims/EmailAddress"
        MellonSetEnvNoPrefix "ADFS_LOGIN" "http://schemas.xmlsoap.org/claims/CommonName"
        MellonSetEnvNoPrefix "ADFS_FULLNAME" "http://schemas.xmlsoap.org/claims/DisplayName"
        MellonSetEnvNoPrefix "ADFS_PHONENUMBER" "http://schemas.xmlsoap.org/claims/PhoneNumber"
        MellonSetEnvNoPrefix "ADFS_FAXNUMBER" "http://schemas.xmlsoap.org/claims/FaxNumber"
        MellonSetEnvNoPrefix "ADFS_MOBILENUMBER" "http://schemas.xmlsoap.org/claims/MobileNumber"
        MellonSetEnvNoPrefix "ADFS_BUILDING" "http://schemas.xmlsoap.org/claims/Building"
        MellonSetEnvNoPrefix "ADFS_FIRSTNAME" "http://schemas.xmlsoap.org/claims/Firstname"
        MellonSetEnvNoPrefix "ADFS_LASTNAME" "http://schemas.xmlsoap.org/claims/Lastname"
        MellonSetEnvNoPrefix "ADFS_DEPARTMENT" "http://schemas.xmlsoap.org/claims/Department"
        MellonSetEnvNoPrefix "ADFS_HOMEINSTITUTE" "http://schemas.xmlsoap.org/claims/HomeInstitute"
        MellonSetEnvNoPrefix "ADFS_HOMEDIR" "http://schemas.xmlsoap.org/claims/HomeDir"
        MellonSetEnvNoPrefix "ADFS_PERSONID" "http://schemas.xmlsoap.org/claims/PersonID"
        MellonSetEnvNoPrefix "ADFS_PREFERREDLANGUAGE" "http://schemas.xmlsoap.org/claims/PreferredLanguage"
        MellonSetEnvNoPrefix "ADFS_ROLE" "http://schemas.microsoft.com/ws/2008/06/identity/claims/role"
        MellonSetEnvNoPrefix "ADFS_IDENTITYCLASS" "http://schemas.xmlsoap.org/claims/IdentityClass"
        MellonSetEnvNoPrefix "ADFS_FEDERATION" "http://schemas.xmlsoap.org/claims/Federation"
        MellonSetEnvNoPrefix "ADFS_AUTHLEVEL" "http://schemas.xmlsoap.org/claims/AuthLevel"
        MellonSetEnvNoPrefix "ADFS_FEDACLS" "http://cern.ch/fedacls"
        MellonEnable "info"
        MellonSPPrivateKeyFile /mnt/data2/etc/httpd.koji/conf.d/mellon/https_koji.cern.ch_mellon.key
        MellonSPCertFile /mnt/data2/etc/httpd.koji/conf.d/mellon/https_koji.cern.ch_mellon.cert
        MellonSPMetadataFile /mnt/data2/etc/httpd.koji/conf.d/mellon/https_koji.cern.ch_mellon.xml
        MellonIdPMetadataFile /mnt/data2/etc/httpd.koji/conf.d/mellon/FederationMetadata.xml
        MellonEndpointPath /mellon
        #MellonUser "CommonName"


5/ Configure protected directory


        MellonEnable "auth"
        MellonSamlResponseDump On 
        MellonUser "ADFS_LOGIN" 


Please Note MellonUser set the rEMOTE_USER for application, so to get the correct username for my application I had to add :