On package signing procedures

Introduction and goal of rpm signing in certification

Our goal is to ensure that rpms which were certified in the certification procedure has not been changed and/or compromised while they arrive to production. A solution to this problem is applying digital signatures to the validated rpms.

GPG support has been added to RPM 4.0, which is suitable for our purpose.

The GPG key

GnuPG is the GNU project's complete and free implementation of the OpenPGP standard as defined by RFC4880.

Generating the key

The gpg utility can be used to generate a key for signing rpms. This utility uses the GPGHOME enviroment variable to determine the place where the keys will be generated. The default value is $HOME/.gnupg which is not really safe, (AFS issues, etc), so for the time of key-generation it is better to overwrite the default settings.

The secure way of generating a key is the following:

1. Deconnect your computer from the network, disable wifi and bluetooth, as well.
2. Create a temporarly directory

           mydir=`mktemp -d /tmp/tmpXXXXXX`
           chmod 700 $mydir
           export GNUPGHOME="$mydir"
           gpg --gen-key
        

   Answer the questions, choose key length 2048 and expiry time 365 days. This has to be decided, since in principle there is no reason for such key to expiry...
3. Back up the key immediately to a USB drive (read the relevant sections)
4. Delete the directories

           rm -rf $mydir
           unset $GNUPGHOME
         

5. If you are paranoid then you absolutely should delete the file from the hard disk, before step (d.). I.e erasing the file physically not only the name space entry. For this purpose you can use for example the 'shred' utility.
6. If you are really paranoid it is better to delete your bash_history. (log out from shell, log in again and delete)
7. If you are even more paranoic than that then you should not store your keys on USB drive, but you should store them on a OpenPGP smartcard.

           http://www.kernelconcepts.de/products/security-en.shtml
         

   This stuff is quite good and will destroy itself after 3 successive password guess, so no brute force techniques could be applied in case of loss.
8. Unmount and remove your USB drive, reboot your computer.
9. Now, you can reconnect your computer to the network.

Generating the revocation certificate

A revocation certificate is necessary if the private key has been compromised or its password lost.

        gpg --output revoke.asc --gen-revoke keyID
     

The generated revocation certificate is short, so the best way is to print it out and keep separately from your private key, but still in a very safe place.

Publishing the key

The public part of the key should be published on the project's website together with it's fingerprint. The key's fingerprint can be obtained by

        gpg --fingerprint

Don't put the public key in the repository. People tend to accept keys only by their location which is a bad practice.

Protecting the key

You have to protect the key against being compromised and also against being lost. You have to protect your key with a strong password. You can use one of the password quality meters to check its strength

        http://www.geekwisdom.com/dyn/passwdmeter

The password should be known only by the release manager and his/her lawyer. To change the password you can use the

        gpg --edit-keys keyID passwd

command.

Backing up the key

Back up the key generated and keep a copy offline !

Export the public part of the key with

        gpg -ao certifkey.pub --export keyID

The key ID can be obtained with the

        gpg --list-keys

command. To backup its private part you can use the command

        gpg -ao certifkey.key --export-secret-keys keyID

or

        gpg -ao certifkey.key --export-secret-keys keyID | gpg -aco certifkey.key

which adds a password to the exported private key.

Back up a paper copy of the private key ! This is not a joke and is much safer then any digital backup solution. Don't print that out on a network print server. Print out at home.

Removing the key

If, for some reason you have to remove the gpg key from your keyring, you can do this with the following command:

      gpg --delete-key keyID

Restoring the key

To restore your keys from your backup, simply import it with

        gpg --import certifkey.key

or if it was encrypted

        gpg --decrypt certif.key.gpg | gpg --import

Restoring the public key in case of loss

All OpenPGP secret keys contain an internal copy of the public key.

        gpgsplit --no-split --secret-to-public certifkey.gpg > certifkey.pub
        gpg --import certifkey.pub

For this the key has to be dearmored, what you can achive with

        gpg --dearmor < certifkey.asc > certifkey.gpg

Restoring the private key in case of loss

If the private key has lost, there is no way to recover it !

Key expiration

For security reasons the generated key is (and should be) valid for 1 year. When it expires it is necessary to generate a new pair of keys.

The new keys can be generated in the same way as previous (see "Generating the key" section), so just to remind in short:

        $ gpg --gen-key     # key generation (setting 1 year valid time, 
                           # and "Gd Integration <gd-release-team@cern.ch>" as info)
        $ gpg -k                # to see your public keys 
        $ gpg -K               # to see your private keys 

Then the public key have to be exported to an ascii file (for distribution):

        $ gpg --armor --export "Gd Integration <gd-release-team@cern.ch>"  > new_glite_key_gd.asc

The key is distributed via web site: http://glite.web.cern.ch/ (this location is specified also in gLite 3.2 Generic Installation Guide, and it should also be present in your yum repository files for gLite).

Since the old key(s) is still needed (as there is a number of RPMs signed with it), it should be preserved in the file (if we do not want to resign all old packages still existing in repository...). Therefore the file with key provided on website should not be replaced with the new one, but the new key should be added to the file, for example like this:

       $ wget http://glite.web.cern.ch/glite/glite_key_gd.asc
       $ cat new_glite_key_gd.asc >> glite_key_gd.asc
 

then glite_key_gd.asc with all keys can be copied to the website (replacing the old one).

The file will provide all keys to verify packages, any missing key (old ones, or the new one) that is available in the file will be imported by the package manager when needed (during installation of a package signed with a not known key).

Notice

all sites should probably be informed about updating the key before using it for releasing any packages signed with it, because (for example) when yum/rpm finds a new key, it tries to import it and asks (!) for confirmation. In such case all automatic update procedures might return error. Solution to this for large infrastructure can be downloading the key and importing it on all machines using rpm (it does not require confirmation, so can be easily automated).

Note about GPG version: keys generated with other version of gnupg (using eg. linux distros other than SL) might not work properly (reason unknown...). In case of having problems try using version available in SL5.

Compromised key

Password is compromised

If the password of the private key has been compromised but not used, one can change the password on the private key, by using

         gpg --edit-key keyID  passwd

Private key is compromised

If there is a chance that your private key is compromised you have to revoke it immediately. If your public key was part of a keyring, then you should post you revocation certificate to the same keyring exactly the same way as you posted your key. The keyring servers are add-only, so there is no way to withdraw a revocation certificate. Once it is posted your key is revoked.

However the problem is that rpm is not aware of public keyrings, it can only use a local keyring database.

Signing rpm packages

Import the key

In order to sign the package you have to import the keys. See the 'Restoring the key' section on how to do this.

Signing a package

After importing the public key to the RPM DB edit the ~/.rpmmacros file and put the following information

        %_signature gpg
        %_gpg_name  <Your Name which is in the gpg key>

Then one can sign already existing packages with

        gpg --resign *.rpm

or sign the rpm package during build with

        rpmbuild -ba --sign SPEC/package.spec

Real paranoid sysadmins use their own version of gpg from a USB drive, i.e. safe against compromised gpg binaries. In this case use the following ~/.rpmmacros file:

       %_signature gpg
       %_gpg_path /mnt/disk/keys/.gpg
       %_gpg_name gLite Certification <glite@foo.com>
       %_gpgbin /mnt/disk/gpg/gpg.exe

Verifying the signatures

When you would like to check digital signature, you need to import the public key into the RPM database:

        wget http://glite.org/certifkey.pub
        rpm --import certifkey.pub
        rpm -K package.rpm

To check that the public key has really been imported you can use the command

        rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'

Removing rpm signature

This is not easy, so before signing create a backup with the original rpm.

Various signing procedures

Normal operation

The current procedure for signing rpms is the following:

1. A certificator receives a patch i.e it is made available for certification
2. A certificator certifies the patch
3. A certificator notifies the signing responsible person (SRP) that rpms are ready to sign
4. The SRP signs the rpms while renewing the repository
5. The SRP checks the signature with the public key published on the web (not the local version)
6. The SRP place the signed rpms to a new repository.
7. The SRP marks the patch as 'signed'
8. The patch is then moved to PPS

Issues:

1. One should minimize the time between 3 and 4
2. The SRP should use the key from an external, removable storage, i.e by resetting GNUPGHOME to a non standard location.
3. The signing should be a manual action with the minimum amount of script involved

Private key has been deleted

If you accidentally delete your private key, and you are sure it is not compromised, then there is no need to revoke the old one, just releasing a new one. The following procedure could be followed.

1. Generate a new key-pair according to the previous sections
2. Publish your new public key

Private key has been compromised

If you accidentally delete your private key, and you are sure it is not compromised, the following procedure could be followed:

1. Pick up the BRT (Big Red Telephone) and inform your boss.
2. If you still have a job continue reading
3. If you posted your public key to a keyring, post your revocation key to the same keyring.
4. Inform your customers and make them erasing your public key from their package manager public key database
5. Generate a new key-pair
6. Publish your new public key

References

1. RPM signing howto
2. GPG HowTos
3. GPG FAQ
4. The Strong Distribution Model

-- GergelyDebreczeni - 27 Jan 2009