DPM schema upgrade to >= 1.7.0

Below are the steps needed to upgrade the DPM database schema from that used between versions 1.6.4 and 1.6.11 to versions >= 1.7.0. (The gLite DPM 1.7.0 release also has a different information provider with respect to 1.6.x releases. The necessary changes to use the new information provider are made by YAIM and are not listed on this page)

For upgrading from DPM < 1.6.4 you should upgrade in steps. See

https://twiki.cern.ch/twiki/bin/view/LCG/DpmSecondaryGroupsSupport

InnoDB pool size for mysql

It has been found that with mysql the schema update may require an InnoDB pool larger than the default. Therefore it is recommended that the InnoDB pool size is first enlarged and then the schema upgrade run. For example add the following to the mysql configuration file (/etc/my.conf) in the [mysqld] section:

set-variable=innodb_buffer_pool_size=256M

and restart mysql. If the InnoDB pool size is insufficient it will give the error "total number of locks exceeds the lock table size" during the schema upgrade, see the Troubleshooting section below on how to proceed. YAIM will not check or change the InnoDB pool size.

Stop all daemons

If you use YAIM to configure your installation everything from here on is done automatically. You don't need to do the following configuration and schema upgrade steps manually. However, as usual, RPM installation or upgrade to the new version is not done by YAIM itself!

In the following order, on the appropriate machines:

• service dpm-gsiftp stop on all disk servers
• service srmv2.2 stop on the SRM host
• service srmv2 stop on the SRM host
• service srmv1 stop on the SRM host
• service dpm stop on the DPM host
• service rfiod stop on all the disk servers
• service dpnsdaemon stop on the DPNS host

Install the new RPMs

All of them, on the appropriate node(s):

• DPM-server
• DPM-copy-server (this is a new RPM in the 1.7 series and is installed on the DPM host)
• DPM-name-server
• DPM-srm-server
• DPM-rfio-server
• DPM-gridftp-server
• DPM-client

Database backup

IMPORTANT

As the migration script modifies the database content, we highly recommend that you take a backup of the DPM databases before running the migration script. Please backup the dpm_db database beforehand, as this is the database which will be changed. It is also recommended to backup cns_db as well, although the migration does not change that database.

Run the migration script

To support some of the new features in DPM 1.7 a database schema change is needed. The migration script is included in the DPM-name-server RPM.

Requirements:

• Perl-DBD (for Mysql or Oracle)
• Perl-DBI

NOTE Depending of the number of entries in the database, the script takes more or less time - possibly up to several hours. So, be patient ! If you have a large dpm_db database, you may consider applying the suggested adjustement in MySQL configuration described in Troubleshooting section about lock table size.

Example for a Mysql DPM

$ cd /opt/lcg/share/DPM/dpm-db-310-to-320
$ ./dpm_db_310_to_320 --db-vendor MySQL --db localhost --user dpm --pwd-file /root/pwd --dpm-db dpm_db --verbose

Example for an Oracle DPM

$ cd /opt/lcg/share/DPM/dpm-db-310-to-320
$ ./dpm_db_310_to_320 --db-vendor Oracle --db devdb --user dpm --pwd-file /root/pwd --verbose

Restart the daemons

• service dpnsdaemon start on the DPNS host
• service rfiod start on all the disk servers
• service dpm start on the DPM host
• service srmv1 start on the SRM host
• service srmv2 start on the SRM host
• service srmv2.2 start on the SRM host
• service dpm-gsiftp start on all disk servers
• service dpmcopyd start on the DPM host

Troubleshooting

total number of locks exceeds the lock table size

If you get this error when running the script on a mysql database:

failed to query and/or update the DPM database : DBD::mysql::db do failed: 
The total number of locks exceeds the lock table size at 
UpdateDpmDatabase.pm line 19.
Issuing rollback() for database handle being DESTROY'd without explicit 
disconnect().

it means your request database (dpm_db) is large and that InnoDB pool size is not large enough to accomodate the transaction required by the database upgrade. To fix the problem, edit the configuration file for MySQL and add the following line in [mysqld] section:

set-variable=innodb_buffer_pool_size=256M

The value 256M is indicative and should be adjusted based on the memory configuration of the mysql server. The larger the value, the better it is for performance, as long as you don't cause paging or swapping. See http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size for more details.

After fixing the MySQL configuration, restart MySQL and re-run the script. Before doing this follow the indication in 'Error when running the scirpt twice'.

NOTE The above change for the InnoDB pool size can be left set permanently and should have a positive impact on performance.

Error when running the script twice

The script should be run only once. If you run it twice, the following error will appear:

$ ./dpm_db_310_to_320 --db-vendor MySQL --db localhost --user dpm --pwd-file /root/pwd --dpm-db dpm_db --verbose
Mon Oct 20 17:11:05 2008 : Starting to update the DPNS/DPM database.
Please wait...
failed to query and/or update the DPM database : DBD::mysql::db do failed: Duplicate column name 'r_uid' at UpdateDpmDatabase.pm line 18.
Issuing rollback() for database handle being DESTROY'd without explicit disconnect().

If you need to rerun the schema upgrade script because of the InnoDB buffer pool size problem described above, apply the following SQL (e.g. with mysql command line tool):

use dpm_db
alter table dpm_get_filereq drop column r_uid;

after which it is possible rerun the schema upgrade script. If you received some other error during the schema upgrade and believe you may need to rerun it please contact dpm-users-forum or hep-service-dpm@cernNOSPAMPLEASE.ch for advice.

-- DavidSmith - 29 June 2009