Koji
User management
Web interface
Connect with firefox :
https://koji.cern.ch/koji/
Authentication is done with Kerberos so you need to add in "about:config" two options:
network.negotiate-auth.delegation-uris = cern.ch
network.negotiate-auth.trusted-uris = cern.ch
Add user account
STEP 1 : koji add-user --principal=<username>@CERN.CH <username>
Please note the user has been created automatically if he tried to login in the web interface so you may get:
GenericError: user already exists: user
You can safely execute the second command to give them the required permission:
STEP 2 : koji grant-permission build <username>
Redirect user to the documentation:
https://twiki.cern.ch/twiki/bin/view/LinuxSupport/BuildingRPMswithKoji
And warn them that it is a
BETA service.
Add admin account
koji add-user --principal=<username>@CERN.CH <username>
koji grant-permission admin <username>
Create new tag
# vi /mnt/data2/home/build/bin/tags/createtag5.sh
# vi /mnt/data2/home/build/bin/tags/createtag6.sh
# vi /mnt/data2/home/build/bin/tags/createtag7-el7.sh
Machines
koji hub: koji.cern.ch
builder: lxdist01 lxdist02 lxdist03 lxdist04
fstab
/dev/mapper/vg_lxsoft06-root / ext4 defaults 1 1
UUID=fa79925c-fcbe-4b8d-8c37-30c0d25f2e2a /boot ext4 defaults 1 2
/dev/mapper/vg_lxsoft06-tmp /tmp ext4 defaults 1 2
/dev/mapper/vg_lxsoft06-var /var ext4 defaults 1 2
/dev/mapper/vg_lxsoft06-swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
UUID=75c859da-fd35-4d96-9770-06aa1e28ef60 /mnt/koji ext4 defaults 1 1
/dev/mapper/mirror-lvol0 /mirror ext4 defaults 1 1
apache
kojira
It regenerate automically koji internal repository when packages are added
- ps -eaf | grep kojira
- check /var/log/kojira.log for errors.
mash
It generate repository for our tags and is used by Hamster/ Agile team
- /root/bin/execute-mash.sh
- executed every 5 minutes
In case of failure
builder kojid
- check if kojid runs
- check logs in /var/log//kojid.log
Signing key management
- backup /mnt/data2/etc/koji-hub/gnupg just in case
- execute:
export GNUPGHOME=/mnt/data2/etc/koji-hub/gnupg
gpg --import public.asc
gpg --import private.asc
- Test that you can sign manually:
rpm --resign --define '_signature gpg' --define '_gpg_name CERN MIG ' --define '_gpgbin /usr/bin/gpg' --define '_gpg_path /mnt/data2/etc/koji-hub/gnupg/' mytest.rpm
rpm -K mytest.rpm
- Verify permision on $GNUPGHOME
# ls -ld $GNUPGHOME
drwx------. 3 apache apache 4096 Feb 26 14:59 gnupg
# ls -l $GNUPGHOME
rw-------. 1 apache apache 7856 Oct 26 2012 gpg.conf
drw-------. 2 apache apache 4096 Oct 26 2012 private-keys-v1.d
-rw-------. 1 apache apache 2342 Feb 26 10:11 pubring.gpg
-rw-------. 1 apache apache 2342 Feb 26 10:11 pubring.gpg~
-rw-------. 1 apache apache 600 Oct 26 2012 random_seed
-rw-------. 1 apache apache 968 Oct 26 2012 RPM-GPG-KEY-agileinf
-rw-------. 1 apache apache 2406 Feb 26 10:03 RPM-GPG-KEY-mig
-rw-------. 1 apache apache 3880 Feb 26 10:17 secring.gpg
-rw-------. 1 apache apache 1280 Feb 26 10:11 trustdb.gpg
- Define the tags you need to sign in /mnt/data2/etc/koji-hub/plugins/sign.conf like this:
[ai6-testing]
rpm = /bin/rpm
gpgbin = /usr/bin/gpg
gpg_path = /mnt/data2/etc/koji-hub/gnupg
gpg_name = CERN agileinf
gpg_pass = ****************
enabled = 1
Common problems
Q: My kernel module is not building anymore, and ask for latest kernel version.
- Check koji builder and upgrade them to latest kernel. using wass on lxadm.
- Check spec file is correct
Q: Tasks stay in purple in web interface and in "Free" state
- Restart kojid on builder /etc/init.d/kojid restart
- Check free space on /build (puppet should take care of that through cron.)
Q: Package X-Y-Z is missing but it seems it should be available in the repo.
- Check if it is tagged properly. Some package are built for tag el6_3 and are not then tagged on el6 e.g : koji tag-pkg el6 expat-2.0.1-11.el6_2 It should be fixed for all package built from mid-july 2012.
- Koji is using a slc6 mirror from linuxsoft.cern.ch in lxsoft06:/mirror/
- Check the package has not been blocked for the specific tag e.g : koji list-pkgs --show-blocked --tag=ai6
Q: This error message appears :
GenericError: hash changed for external rpm:
autoconf-2.59-12.noarch@el5-external-repo-os-2 (00f623da00db01162455899eecd744e2 -> de86fbcf57041ef7e0695783699fce1a)
Q: How to push updates from Koji to Linuxsoft?
Add a Koji administrator to run:
Q: I have strange auth error in the logs
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] 2013-11-19 09:32:50,874 [WARNING] m=listTasks u=None p=8845 r=137.138.144.42:60050 koji.xmlrpc: Traceback (most recent call last):
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] File "/usr/share/koji-hub/kojixmlrpc.py", line 212, in _wrap_handler
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] response = handler(environ)
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] File "/usr/share/koji-hub/kojixmlrpc.py", line 255, in handle_rpc
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] return self._dispatch(method, params)
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] File "/usr/share/koji-hub/kojixmlrpc.py", line 279, in _dispatch
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] self.check_session()
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] File "/usr/share/koji-hub/kojixmlrpc.py", line 261, in check_session
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] context.session = koji.auth.Session()
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] File "/usr/lib/python2.6/site-packages/koji/auth.py", line 112, in __init__
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] raise koji.AuthError, 'Invalid session or bad credentials'
Nov 19 09:32:50 lxdist01 httpd[8845]: [error] AuthError: Invalid session or bad credentials
Nov 19 09:32:50 lxdist01 httpd[8845]: [error]
- Verify that "kojira" run on the correct node in the cluster.
- clustat | grep koji
- kill kojira on other nodes.
Q: I strange python error in the root.log "Not using downloaded repomd.xml because it is older than what we have"
Executing command: /usr/bin/repoquery -c /var/build/mock/el6rt-build-44358-31748/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/build/mock/el6rt-build-44358-31748/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
Not using downloaded repomd.xml because it is older than what we have:
DEBUG util.py:264: Current : Tue Apr 29 10:41:52 2014
DEBUG util.py:264: Downloaded: Thu Apr 24 14:54:54 2014
DEBUG util.py:264: Traceback (most recent call last):
DEBUG util.py:264: File "/usr/bin/repoquery", line 1241, in
DEBUG util.py:264: main(sys.argv)
DEBUG util.py:264: File "/usr/bin/repoquery", line 1235, in main
DEBUG util.py:264: repoq.runQuery(regexs)
DEBUG util.py:264: File "/usr/bin/repoquery", line 805, in runQuery
DEBUG util.py:264: pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs)
DEBUG util.py:264: File "/usr/bin/repoquery", line 742, in matchPkgs
DEBUG util.py:264: pkgs = self.returnPkgList(patterns=items)
DEBUG util.py:264: File "/usr/bin/repoquery", line 695, in returnPkgList
DEBUG util.py:264: pkgs = self.pkgSack.returnNewestByNameArch(**kwargs)
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 887, in
DEBUG util.py:264: pkgSack = property(fget=lambda self: self._getSacks(),
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 669, in _getSacks
DEBUG util.py:264: self.repos.populateSack(which=repos)
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/repos.py", line 308, in populateSack
DEBUG util.py:264: sack.populate(repo, mdtype, callback, cacheonly)
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 165, in populate
DEBUG util.py:264: if self._check_db_version(repo, mydbtype):
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 223, in _check_db_version
DEBUG util.py:264: return repo._check_db_version(mdtype)
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1256, in _check_db_version
DEBUG util.py:264: repoXML = self.repoXML
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1455, in
DEBUG util.py:264: repoXML = property(fget=lambda self: self._getRepoXML(),
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1447, in _getRepoXML
DEBUG util.py:264: self._loadRepoXML(text=self)
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1437, in _loadRepoXML
DEBUG util.py:264: return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1412, in _groupLoadRepoXML
DEBUG util.py:264: if self._commonLoadRepoXML(text):
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1250, in _commonLoadRepoXML
DEBUG util.py:264: self._revertOldRepoXML()
DEBUG util.py:264: File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1083, in _revertOldRepoXML
DEBUG util.py:264: os.rename(old_data['old_local'], old_data['local'])
DEBUG util.py:264: OSError: [Errno 2] No such file or directory
- run
koji regen-repo XYZ-build
, with XYZ being the repo in question (can be found in the first line of the log, for this example el6rt)
SSO configuration
0/ Verify you have a valid SSL certificate
https://gridca.cern.ch/gridca/
1/ Register your application
http://www.cern.ch/sso-management
Register New SSO Application
Application name: https://koji.cern.ch
Application URI: https://koji.cern.ch/mellon
Service provider: SAML2
Application homepage: https://koji.cern.ch
Application description: koji
2/ Get Saml certificate:
cd mnt/data2/etc/httpd.koji/conf.d/mellon
/usr/libexec/mod_auth_mellon/mellon_create_metadata.sh https://koji.cern.ch/mellon https://koji.cern.ch/mellon
you will get three files:
https_koji.cern.ch_mellon.cert
https_koji.cern.ch_mellon.key
https_koji.cern.ch_mellon.xml
3/ Get Metadata
wget https://login.cern.ch/FederationMetadata/2007-06/FederationMetadata.xml
(or wget https://login-dev.cern.ch/FederationMetadata/2007-06/FederationMetadata.xml)
4/ Serve the xml file on https://koji.cern.ch/metadata
mkdir /var/www/html/metadata
cp https_koji.cern.ch_mellon.xml /var/www/html/metadata/koji.xml
Verify that https://koji.cern.ch/metadata/koji.xml is correctly served.
5/ Configure Apache
MellonSetEnvNoPrefix "ADFS_GROUP" "http://schemas.xmlsoap.org/claims/Group"
MellonSetEnvNoPrefix "ADFS_EMAIL" "http://schemas.xmlsoap.org/claims/EmailAddress"
MellonSetEnvNoPrefix "ADFS_LOGIN" "http://schemas.xmlsoap.org/claims/CommonName"
MellonSetEnvNoPrefix "ADFS_FULLNAME" "http://schemas.xmlsoap.org/claims/DisplayName"
MellonSetEnvNoPrefix "ADFS_PHONENUMBER" "http://schemas.xmlsoap.org/claims/PhoneNumber"
MellonSetEnvNoPrefix "ADFS_FAXNUMBER" "http://schemas.xmlsoap.org/claims/FaxNumber"
MellonSetEnvNoPrefix "ADFS_MOBILENUMBER" "http://schemas.xmlsoap.org/claims/MobileNumber"
MellonSetEnvNoPrefix "ADFS_BUILDING" "http://schemas.xmlsoap.org/claims/Building"
MellonSetEnvNoPrefix "ADFS_FIRSTNAME" "http://schemas.xmlsoap.org/claims/Firstname"
MellonSetEnvNoPrefix "ADFS_LASTNAME" "http://schemas.xmlsoap.org/claims/Lastname"
MellonSetEnvNoPrefix "ADFS_DEPARTMENT" "http://schemas.xmlsoap.org/claims/Department"
MellonSetEnvNoPrefix "ADFS_HOMEINSTITUTE" "http://schemas.xmlsoap.org/claims/HomeInstitute"
MellonSetEnvNoPrefix "ADFS_HOMEDIR" "http://schemas.xmlsoap.org/claims/HomeDir"
MellonSetEnvNoPrefix "ADFS_PERSONID" "http://schemas.xmlsoap.org/claims/PersonID"
MellonSetEnvNoPrefix "ADFS_PREFERREDLANGUAGE" "http://schemas.xmlsoap.org/claims/PreferredLanguage"
MellonSetEnvNoPrefix "ADFS_ROLE" "http://schemas.microsoft.com/ws/2008/06/identity/claims/role"
MellonSetEnvNoPrefix "ADFS_IDENTITYCLASS" "http://schemas.xmlsoap.org/claims/IdentityClass"
MellonSetEnvNoPrefix "ADFS_FEDERATION" "http://schemas.xmlsoap.org/claims/Federation"
MellonSetEnvNoPrefix "ADFS_AUTHLEVEL" "http://schemas.xmlsoap.org/claims/AuthLevel"
MellonSetEnvNoPrefix "ADFS_FEDACLS" "http://cern.ch/fedacls"
MellonEnable "info"
MellonSPPrivateKeyFile /mnt/data2/etc/httpd.koji/conf.d/mellon/https_koji.cern.ch_mellon.key
MellonSPCertFile /mnt/data2/etc/httpd.koji/conf.d/mellon/https_koji.cern.ch_mellon.cert
MellonSPMetadataFile /mnt/data2/etc/httpd.koji/conf.d/mellon/https_koji.cern.ch_mellon.xml
MellonIdPMetadataFile /mnt/data2/etc/httpd.koji/conf.d/mellon/FederationMetadata.xml
MellonEndpointPath /mellon
#MellonUser "CommonName"
5/ Configure protected directory
MellonEnable "auth"
MellonSamlResponseDump On
MellonUser "ADFS_LOGIN"
Please Note MellonUser set the rEMOTE_USER for application, so to get the correct username for my application I had to add :