Metadata Access from the Shell

You can access the metadata catalogue either with the mdclient metadata terminal tool as configured in the last section, or via the mdcli (or mdjavacli for those using the java package) command line tool which allows you to directly issue metadata commands on the shell, it's output is intended to be easily parseable by scripting languages:

  $ mdcli -p8822 -slocalhost listattr /
  t
  text
  f
  float

Metadata commands are parsed into pieces which are each separated by white space similarly to shell commands. If you want the white space to be part of one piece of the command itself, for example when you want to set an attribute to a string which contains white space, you must enclose it in singe quotes: ' '. Single quotes are part of the command syntax and used when parsing the commands into parts. You need them every time a part shall contain spaces. Double quotes however are used in queries Metadata Queries (expressions evaluated by the database backend) to distinguish strings from variable references and common values. In order to put a single quote into a command argument or any other character, you can use an octal code, e.g. to get a "'".

Note that quotes may be removed by the shell when parsing a shell command, so if you are using the mdcli tool where the AMGA command is given on the command line, you will need to protect these single quotes from being removed by the shell with double quotes:" ". The various APIs will in contrast usually (that is the Python and Java APIs do so, but not the C-API) automatically quote any arguments you pass to them with single quotes so they are not to be used in those APIs. The following is an example with mdclient, mdcli and the Python API showing how quotes are being used:

  $ mdclient
  Query> find /files '/files:producer="CERN"'

  $ mdcli find /files "'/files:producer="CERN"'"

An in python:

  mdclient.find('/files', '/files:producer="CERN"');

The metadata server uses a streaming protocol. Some APIs (for example the Java one) allow to interrupt the streaming of a response. The same is true for the mdclient. Pressing CTRL-C once during the transmission of the result will interrupt the streaming of the result. Only pressing CTRL-C a second time will terminate the client.

In the following is given a list of metadata commands. Additionally commands may be available for group or user access management, as described in Users, Groups and ACLs or Management of Users using the database backend depending on the server setup. To find out which commands are available on the server you are connected to, use the help command.

Manipulating Collections

• createdir /parentdir/dir [option]: Creates the directory dir if it does not yet exist but parentdir already exists. The directory is created with the current owner and the same ACLs as the parent directory. The option filed is a comma seperated list of options (no spaces allowed). The following options are available as of AMGA 1.1:
  • shared: Subdirectories created under this directory share the same schema and database table of the parent directory
  • acls: Creates a directory with acls for every entry, this is currently only supported by PostgreSQL and MySQL5 and only if the necessary supporting stored procedures have been installed first.
  • type=<type> Specifies the data type of the entry column. Explicitely supported are int, float, date.
  • table=<TableType>: Only MySQL. Allows to specify the storage type of the table, e.g. InnoDB. Neccessary because not all tables are alike. Specify e.g. MyIsam, this allows for example GIS functionality.

• ls [options] [directory]: Returns the name of all subdirectories, files, indexes, sequences, and views in the directory in a very similar way to the UNIX ls command. The following options can be used with ls.
  • l : use a long listing format
  • s : returns subdirectories only

• dir [directory]: Returns the name of all subdirectories and files in the directory. The result is returned in the following way:

      0
      entry 1
      entry-type-1
      ...
      entry n
      entry-type-n
      EOT
  

  where the entry-type is either 'entry' or 'collection'. If AMGA collaborates with a file catalogue this command will effectively show the content of the file catalogue. If you want to see which entries have already been attached to a schema in the AMGA part use the listentries command.

• stat [dir/entry]: Returns information on a given entry or directory. They can also be a pattern in the case of several entries. You need read permission to get this information. For an entry the following is returned (if the information is not stored in the table, it

      0
      name
      type (=entry)
      owner-permissions
      group-permissions
      owner
      size
      creation time (Format: 1999-12-22 04:05:06)
      guid
      expires
      link
  
      EOT
  

  For a directory the following is returned:

      0
      name
      type [options]
      owner-permissions
      owner
      acls
      EOT
  

• rmdir path: Removes all directories matching path. Directories are only deleted if they are empty and they have no attributes defined.

• pwd: Prints the current directory which you can change with cd.

• cd path: Changes the directory to the given path. Possible errors are:
  • 1 No such file or directory
  • 4 Permission denied

Commands for Manipulating Attributes

• addattr dir (key type)+: Adds new keys to the list of keys of a directory. In a relational database backend these keys become the columns of a table associated to a directory. You should only use one key/type pair currently for compatibility reasons because some older backends like PostgreSQL <=7.4 do not allow to alter a table adding several columns. Possible types are explained in AMGA data types. The type is only used as a hint for the back end to store the data efficiently and allow efficient queries. The type may be ignored by the implementation (e.g. if the back end is a filesystem). In a filesystem the types and defined keys could be stored as attributes of directories. Some storage backend may allow you to define keys on a per-entry basis. Possible errors are:
  • 1 No such file or directory
  • 4 Permission denied. You need write permission on the directory.
  • 7 Illegal Key: Keys must be alphanumeric. The following keys are reserverd: ACL, CREATED, FILE, GROUP_RIGHTS, GUID, LINK, MD5, OWNER, PERMISSIONS, SIZE.
  • 9 Internal error: All kinds of errors like duplicate keys
  • 23 Not a directory. You can only add attributes to a directory.

• removeattr dir/entry key+: Removes the attributes or keys from the list of attribtues of the directory dir or the directory of the given entry, or if the implementation allows it from the list of attributes of a given entry. Attributes can only be removed if they are not used by any entry. So you either have to remove all entries for which this key is set or use clearattr to set the value of the attributes to NULL In order to remove an attribute, write permissions on the directory are necessary. Possible Errors are:
  • 1 No such file or directory
  • 4 Permission denied. You need write permission on the directory.
  • 10 No such key
  • 14 Attribute in use

• schema_create dir (attr type)+ [option]: Createds a new directory with a given schema. This is an atomic replacement for a sequence of createdir and addattr. The meaning of the optional option argument is backend dependent and you should not use it if you want to retain this independence. With a MySQL backend you can give here the name of the table engine, for PostgreSQL the keyword inherit will make the table inherit its schema from the parent directory.

• setattr file (attribute value)+: Sets a list of attributes of a file to given values. The attributes must exist.

• getattr pattern (attribute)+: Returns the filename and all attributes in turn for every file matching pattern The following output is returned on success:

      0
      file 1
      attr 1
      ...
      attr n
      file 2
      attr 1
      ...
      attr n
      ...
      file n
      attr 1
      ...
      attr n
      EOT
  

• listattr file: Returns a list of all attributes and their types in the following format:

      0
      attr 1
      type 1
      attr 2
      .
      .
      .
      type n
      EOT
  

• clearattr path attribute: Sets the attributes of all files matching path to NULL. Path may currently contain wildcards only in the file-part. On success returns 0.

Commands for entry manipulation

• addentry entry (attr value)+: Creates a new entry and assigns the given values to the provided attributes. Examples:

      Query> addentry /testdir/a id 10
      Query> addentry b id 10 finished 'Oct-10-2004'
  

  Possible errors are:
  • 3 Illegal command: Syntax error
  • 4 Permission denied: You need write permission on the directory to create a file in it.
  • 7 Illegal Key
  • 10 No such key
  • 15 Entry exists

• insert entry (attr value)+: Creates a new entry and assigns the given evaluated values to the provided attributes. Same as addentry, except that the values are evaluated first, similar to an INSERT statement in SQL.

• addentries (entry)+: Creates the given entries in the catalogue. This command is done in a transaction, that is either all entries are inserted or none. Entries can be spread over several directories. Example:

      Query> addentries a
      Query> addentries /test1/a /test2/a /test1/b
  

  Possible errors are:
  • 1 No such file or directory: You tried to insert into a directory which did not exist.
  • 3 Illegal command: Syntax error
  • 4 Permission denied

• rm [-options] pattern [condition]: Removes all files matching pattern, where pattern may only contain wild cards in the file part. If a condition is given, than that condition needs to be met by the entry's metadata. In order to remove an entry you need write permissions on the parent directory. The pattern can also be the name of a directory if a condition is given, if the table is a plain table, the pattern _must_ be a directory name and there must be a condition given. Valid flags are: -r (recursively delete). Possible errors are:
  • 1 No such file or directory: You tried to insert into a directory which did not exist.
  • 4 Permission denied
  • 20 Not an entry.
  • 29 Operation not permitted on plain table.

• listentries [directory/schema]: Returns the name of all entries in a given directory or schema. This differs from the dir command in that it will not show any directories but also that it shows only entries attached to a schema in the case of an AMGA catalogue collaborating with a file-catalogue. The result is returned in the following way:

      0
      entry 1
      ...
      entry n
      EOT
  

• transaction: Starts a transaction. Any changes to the backend of AMGA are done only when commited. To cancel a transaction use abort.

• upload dir (attribute)+: Starts an upload of entries into the catalogue. Currently a static restriction of the prototype is that there can be only up to 98 attributes assigned like this. After the upload is initialized, the put, abort and commit commands are allowed. Errors are returned by the call immediately, the OK code is delayed till the entire upload is successfully commited.

• put file (values)+: Inserts a new entry during upload. Errors are returned by the call immediately, OK is delayed until upload is commited.

• abort: Aborts upload or transaction. Errors and OK are returned by the call immediately.

• commit: Commits upload or transaction. Errors and OK are returned by the call immediately.

Finding and Updating Entries

• find path query: Returns a list of filenames matching path and fulfilling the query with their attributes. The path may currently contain only wild cards in the file name part. Query must be enclosed in single quotes. Strings in the query must be quoted with double quotes. For the supported syntax see the chapter about Metadata Queries. WARNING: Be careful with patterns which also match a subdirectory, the result is undefined. The result is returned in the following way:

      0
      file 1
      .
      .
      .
      file n
      EOT
  

  The following errors may occur: 8 Illegal query: The query could not be parsed or violates security rules. No further information is currently provided on the reasons for this error. 2 Connection to DB failed or syntax error in path or query

• updateattr pattern (attribute value)+ condition: Updates attributes of entries matching a pattern in a single collection based on a condition. The values to which the attributes are updated can contain attributes as variables. Complex expressions are allowed as values. The condition may reference attributes of other collections. Updates are atomic. Examples:

      Query> updateattr /testdir1/* events events+1 'events>100'
      Query> updateattr /testdir1/* events events+1 '/testdir2:key > 0'
  

  The first example increases the number of events of every file in /testdir1 which has more than 100 events by one. The second example increases the number of events of every file in /testdir1 provided there is an entry in the collection /testdir2 which has the attribute "key" set to anything larger than 1 (usefull to do locking by clients: putting such an entry into /testdir2 would lock /testdir1). For the supported syntax for the queries see the chapter about Metadata Queries.

• updateattr_single pattern (attribute value)+ condition: Almost same as updateattr, but this updates attributes of a random single entry matching a pattern in a single collection based on a condition. This command is useful when AMGA is used as a placeholer for task retrieval. Example:

      Query> updateattr_single /task status "running" 'status="waiting"'
      >> /task/simulration-005
  

• update pattern (attribute value)+ condition: Same as updateattr, but the values are not evaluated prior to insertion into the table. This command works with bound variables, which will also fix problems with SQL command length limitations in Oracle.

• selectattr (attribute)+ condition: Selects attributes from several collections based on a condition doing an inner join on the collections based on a join condition. The FILE attribute is used to select the entry name of an entry. Example:

      Query> selectattr /jobdir:FILE /configdir:id /jobdir:eventGen /configdir:id
          '/jobdir:events>1000 and /configdir:key=/jobdir:key'
  

  This selects the entry-name of a job, the id in the configuration, the event generator name of a job and the id in a configuration for all jobs and configurations where the job has more than 1000 events and the keys attributes of the jobs and configurations match. For the supported syntax for the queries see the chapter about Metadata Queries.

As of AMGA 1.2 selectattr also supports constraints to the query similar to a SELECT clause. Queries can now take the form:

    query [distinct] [limit xx [offset yy]] [order exp] [group_by exp]

where the distinct keyword translates into a SELECT DISTINCT, the limit and offset clause limits the number of rows returned and with the order clause rows can be ordered according to the given expression. The group_by construct corresponds to the GROUP BY SQL statement.

Native SQL Query

Supporting native SQL query greatly eases the work needed to port existing SQL-based database applications to the Grid using AMGA and complements the existing metadata query language in AMGA.

The followings are currently supported commands for native SQL queries.

• SELECT
• INSERT
• UPDATE
• DELETE

They conforms to the SQL-92 Entry Level <direct_data_statements>. All the keywords in native SQL queries should be provided in capital letters. However table names, aliases, and column names can be either of capital or small. For the details, refer Native SQL Queries .

Permission Handling

• whoami: Prints out the name of the current user. Note that this command does not need any connections of the AMGA server and can thus be also used to do a test on whether an AMGA server is alive and what response time it has.

• chown [-R] entry/dir new_owner: Changes the owner of a directory or entry. Only the owner of an entry is allowed to execute this, or the root-user. chown does not check whether the user exists, since user management is considered to be handled outside of AMGA (ideally). [-R] option changes owner of all the subdirectories recursively. Possible errors are:
  • 1 No such file or directory
  • 4 Permission denied

• chmod entry/dir new_permissions: Changes the access permissions of an entry or directory. Entries have owner and group-permissions, while directories have owner permissions and group permissions are handled via ACLs. Group permissions for entries allow you to remove priviliges granted for all entries in a directory via the directories ACLs. The format of new_permissions is rwxrwx for entries and rwx for directories where "-"-signs can be substitued for the letters if you do not want to give a certain privilege. The permissions for entries are the concatanation of first the user and then the group rights. The x-Flag allows a user to enter a directory or respectively list an entry. r-and w-flags allow users to read/write metadata while the w-flag for directories allows users to create or delete entries in the given directory. Users cannot list directories for which they don't have read permissions. The command works also for patterns and uses a transaction. Possible errors are:
  • 1 No such file/directory
  • 4 Permission denied

Capabilities

Capabilities are additional attributes assigned to individual users. They are used for example to allow a user to replicate login information. Currently no mapping of VOMS capabilities is done, but this could be a future use-case.

• capabilities_add <user> <capability>: Adds the given capability to the user's capabilities. Only root can do this.

• capabilities_remove <user> <capability>: Removes the given capability from the user's capabilities. Only root can do this.

• capabilities_list [user]: Lists all capabilities of a user, default is the current user.

Index Management

• index_create name dir 'key1...keyn' [type]: Creates an index name on a collection directory using several attributes and a given algorithm. Algorithms depend on the backend. The index is later referred to /collection/name in index_remove.

• index_remove /path: Removes an index.

• index_list /path: lists all indexes.

Sudo

Since AMGA 2.0 a sudo command exists, which allows the root user to become any other user. The syntax is: sudo <user>

Table Constraints

• constraint_add_not_null directory attribute name: Adds a not NULL constraint for the given attribute of the directory. Name is the name used to refer to the name of the constraint. It must be unique for that directory. Write permissions on the directory are necessary for this operation.

• constraint_add_unique directory attribute name: Adds a UNIQUE constraint for the given attribute of the directory. Name is the name used to refer to the name of the constraint. It must be unique for that directory. Write permissions on the directory are necessary for this operation. NOTE: On MySQL you can use attribute(length) to set the lenght of indexed columns.

• constraint_add_reference directory attribute reffered_attr name: Adds a foreign key constraint for the given attribute of the directory. The foreign key is given by the referenced attribute which must fully qualify that attribute including the table part, e.g. /dir:attr. Name is the name used to refer to the name of the constraint. It must be unique for that directory. Write permissions on the directory are necessary for this operation.

• constraint_add_check directory check name: Adds a check constraint to the directory. Check constraints are boolean expression which must be true for all entries inserted into the directory. An example would be events > 0 requiring the value assigned to the events attribute to be positive. Name is the name used to refer to the name of the constraint. It must be unique for that directory. Write permissions on the directory are necessary for this operation.

• constraint_add_primary_key directory key(s): Adds primary key constraint of the given directory, which by default is on the entry name only. Example:

      Query> constraint_add_primary_key . t,i
  

  In the given example a the primary key becomes the pair of the t, and i attributes.

• constraint_drop directory p_key: Drops the constraint with the given name from the directory. Write permissions on the directory are necessary for this operation. The primary key of a table can be removed by specifying the name "p_key".

• constraint_list directory: Prints all constraints of a directory. You need read permissions on the concerned directory.

Views

Views allow you to create virtual new tables (directories) that combine the information of other tables, similar to what selectattr does. AMGA uses the native support of the database to provide views, so the actual behaviour depends on the database backend. For example some backends (like PostgreSQL) allow you to update an existing view, which actually updates the tables behind it.

An important use-case of views are to support access restrictions to attributes (the columns of the underlying table). This is a typical use-case for views also in normal databse usage.

Views can be accessed and deleted like normal directories.

• view_create name maindir attr_1 ... attr_n condition: Creates a view with the given name based on the entries in the given directory, attaching the attribtues given in the list to these entries, based on the join condition.

In the following examples, the first one shows a use case where a view (view1) is created using all the entries in the current directory, but using only the attr1 or attr2 columns. After assigning the right permissions to the resulting new directory (./view1), this can be made readeable for users who need to read these attributes, while they will not have access to the resto of the attibutes in the . directory. In the second example a view (view2) is created combining attributes from the current directory and the dir subdirectory.

  Query> view_create view1 . attr1 attr2 ''
  Query> view_create view2 . attr1 ./dir:attr2 'dir:FILE = FILE'

Sequences

Sequences allow the creation of a sequence of integer numbers, which are guaranteed to be unique. They are also monotonically increasing at least during a single AMGA connection. The exact implementation depends on the database backend, which can optimize handing out parts of the sequence in batches, so that two consecutive connections not necessarily get first a smaller number in the sequence and then the larger. Sequences are not supported by MySQL <5.0 and SQLite. On MySQL and Oracle sequences are implemented through stored proceedures. In PostgreSQL the native mechanism is used.

Sequences behave like another directory in a directory. They cannot be deleted with rmdir, howerver, instead sequence_remove must be used. The name of the sequence must be lower case due to limitations in some backends.

• sequence_create name dir [increment] [start value]: Creates a new sequences with the given name in the given directory. The name of the sequence is then /dir/name. It is possible to define the increment as well as the start value. Note that backends may not necessarily follow this behaviour strictly if multiple connections are being used.
• sequence_next sequence: Gets the next value from a sequence.
• sequence_remove sequence: Deletes a sequence.

Backing Up Data

• dump [-sec_all|sec_none] [dir]: Recursively dumps the contents of a directory and all subdirectories so that they can be recreated by calling the sequence of AMGA commands printed out. If no directory is specified, it uses "/", making a full catalogue dump. The first option controls whether entry permissions and ACLs are included in the dump: -sec_all includes them, while -sec_none only dumps the metadata.
  Only root is allowed to use this command. Possible errors are:
  • 4 Permission denied

• user_dump [dir]: Dumps the contents of a the user database such that it can be recreated from the sequence of AMGA commands printed out.
  Only root is allowed to use this command. Possible errors are:
  • 4 Permission denied

• grp_dump [dir]: Dumps out the information on all existing groups so that they can be recreated by calling the sequence of AMGA commands printed out.
  Only root is allowed to use this command. Possible errors are:
  • 4 Permission denied

Site management

For using replication, each AMGA server needs to know about the other servers that take part in the system, in order to communicate with them. These will be refered to as <it>sites</it>. The information about other sites is stored in the backend. Sites have the following configuration properties:

    id
    name
    hostname
    port
    login
    password
    use_ssl
    authenticate_with_certificate
    cert_file
    key_file
    use_grid_proxy
    verify_server_cert
    trusted_cert_dir
    require_data_encryption

id is a numeric identifier internal to each AMGA instance and generated automatically by AMGA when the site is inserted in the configuration. name is an human-readable identifier of the site, which can be freely chosen by the administrator. hostname and port is the network address of the remote site. The rest of the properties control the security settings of the connection to the master and are similar to the ones defined in the mdclient.config file, having a similar meaning. Section Configuration of the C++ and Java command line clients describes their usage.

The following commands can be used to manage the sites and their configuration:

• site_list: Lists all sites and their IDs. Shows only a summary for each site, consisting of id, name, hostname and port.

• site_add site_name [hostname:port]: Registers a site. Only root can perform this operation.
  • 4 Permission denied

• site_remove site: Removes a site. Only root can perform this operation.
  • 4 Permission denied

• site_set_properties site property value [property value]*: Updates one or more configuration properties of a site. Only root can perform this operation.
  • 4 Permission denied

• site_get_properties site property [property]*: Gets one or more configuration properties of a site. Only root can perform this operation.
  • 4 Permission denied

• site_list_properties site: List all configuration properties of a site, as a list of key/value pairs. Only root can perform this operation.
  • 4 Permission denied

• site_dump [siteName]*: Outputs the configuration of sites as a list of metadata commands that can be executed by AMGA to recreate the configuration. If no siteName is provided, it dumps all sites in the configuration. Only root can perform this operation.
  • 4 Permission denied

Various administrative commands

• import tablename dir: Makes the table given by tablename available under the given dir as a plain table. Only root can perform this operation.
  • 4 Permission denied

• execute script [Options]...: Executes a script in the directory given by the ExecRoot option in the server configuration file. The script is run with the user-id given in the ExecUser configuration variable, if the amgad is started as root. The script has access to the USER and DN environment variables giving the user name of the amga user and the distinguished name if a certificate was used to authentiate to AMGA. To enable this feature, ExecRoot must be set. The administrator of AMGA is responsible to set this up carefully and make sure that scripts are checking the permissions in USER/DN. The stdout of the script is returned through AMGA to the client. This gives AMGA a functionality similar to a cgi-bin in a web-server. The following error codes exist:
  • 21 Function not implemented: ExecRoot not set.
  • 4 Permission denied: Cannot execute anything outside of ExecRoot.

• backend: Returns the name of the current database backend.
• version: Returns a version number of the connected AMGA server.

Replication

The following are the commands used to control replication from the AMGA command line interface. Some of them are for nodes acting as slaves, others for nodes acting as masters. Nodes acting both as slave and master can use all of them. Section Replication in AMGA provides the background information required to understand these commands.

Commands for Slave Nodes

Slave nodes are responsible for initiating replication, by contacting the master and requesting the replication of the directories they are interested on. This is done using the following commands:

• rep_list_mounts: Lists all local mounts and their current state. It prints out the following information for each mount

      <nodeID>:<directory> - <currentXID>, <state>
  

  where:
  • <nodeID> is the master from where the directory is being replicated.
  • <directory> is the root directory of this mount.
  • <currentXID> is the xid of the last log that was applied to this mount
  • <current state> is either Disconnected if there is no connection to this master, Reconnecting (after a broken connection, trying to reconnect to master) or Receiving (connected to master and receiving logs).

• rep_mount [-nosync | -noperms] nodeID remoteDirectory: Mounts remoteDirectory from the master with id nodeID. After mounted, remoteDirectory is used to refer to this mount and is therefore also called mountPoint. This command fails if remoteDirectory already exists locally or if its parent directory does not exist. For instance, to mount /a/b, directory /a must be created and /a/b should not exist. It also synchronizes the local directory by coping the contents of the mount point from the master to the slave, but it does not make the slave start to receive logs. For that, use rep_start_receive. After this command executes successfully, the mount is on the Inactive state. By default, the permissions and acls associated with the metadata are also replicated. If -noperms is specified, the metadata will be replicated without any security information. If -nosync option is specified, existing metadata is not synchronezed but changes are replicated to the slave after rep_mount. On the slave, the metadata will be owned by the local user performing replication (which currently must be root).

• rep_umount [-noerase] mountPoint: Unmounts the given mount point. The local contents of mountPoint is deleted, as well as the root directory of the mount point. This command only works if the connection to this master is inactive, that is, it should not be receiving logs. See rep_stop_receive. If -noerase option is specified, the existing metadata are not erased with res_umount. The nodeID has to be assigned via site_add with the port of the site pointing to the replication daemon, and where at least the login site-property has to be set.

• rep_mount_users nodeID: Replicates user and group information from the given master. The root user and any groups owned by root are not replicated. The slave must not have any user or group defined, except root or groups owned by root. Each slave can only replicate users and groups from a single master. Apart from these restrictions, the replication of users and groups is handled by any other mount.

• rep_umount_users: Stops replicating users and deletes all users and groups imported from the master. This command only works if the connection to the master is inactive, that is, it should not be receiving logs. See rep_stop_receive.

• rep_users_allow user: Allows the given user to replicate user (login) information. This needs to be allowed done on the master as well as on the slave.

• rep_users_disallow user: Disallows the given user to replicate user information

• rep_start_receive master: Connects to master starts receiving the replication logs for all mount points originating from this node. This opens a TCP connection that is kept open while the slave is running. If the slave is shutdown, the next time it restarts it will try to reestablish the connection to the master to continue receiving logs. The state of all mounts from this master is changed to Receiving.

• rep_stop_receive master: Stops receiving logs from master. The TCP connection is closed and the state of all mount points originating from this node is changed to Disconnected. In this state, connections will not be reestablished when the slave is restarted.

Commands for Master Nodes

The main responsibility of master nodes is to configure the access control rights of slaves. Slaves connect to the master using the standard AMGA users and authenticate in the same way. Access control is done using the replication right, which is granted to users to control the directories they are allowed to replicate. The following commands allow granting and removing this right:

• rep_allow directory user: Grants to user the right to replicate directory and all its subdirectories. Having this permission is a necessary and sufficient condition for a user to be allowed to replicate a directory. In particular, the normal user permissions play no role in deciding to allow a user to replicate a directory.

• rep_disallow directory user: Remove from user the right to replicate directory and all its subdirectories.

• rep_users_allow user: Allows the given user to replicate user (login) information. This needs to be allowed done on the master as well as on the slave.

• rep_users_disallow user: Disallows the given user to replicate user information.

• rep_show_permissions: Show the replication rights for all directories.

• rep_list_subscribers: Show the list of the current subscribers, with their subscriptions. This includes information about whether users/groups and permissions are being replicated and the last xid acknowledge by the subscribers.

AMGA also has special commands for user and group management. They are optional and may not be available on your installation for example if it collaborates with a file catalogue and uses the permission system of that catalogue. For more information see Users, Groups and ACLs and User Management.

Federation

The following are the commands used to control federation from the AMGA command line interface. Section Federation in AMGA provides the background information required to understand these commands. One AMGA server may initiate federation, by contacting other AMGA server and requesting the federation of the directories it is interested on. This is done using the following commands:

• fed_list_mounts: Lists all local mounts and their current state. It prints out the following information for each mount

     Query> fed_list_mounts
      <directory1>
      <site1>
      <directory1>
      <site2>
      ...
  

  where:
  • <directory> is the root directory of this mount.
  • <site> is the site name connected to this mount

• fed_mount site directory: Mounts directory from other AMGA server named with site. After mounted, directory is used to refer to this mount and is therefore also called mountPoint. This command fails if remoteDirectory already exists locally or if its parent directory does not exist. For instance, to mount /a/b, directory /a must be created and /a/b should not exist.

• fed_umount directory: Unmounts the given mount point.
• fed_policy : shows current policy about federation. The first line shows whether connection failures are allowed in federation, and the second line shows whether execution failures are allowsed.
• fed_check_failure shows whether last command had failures. The first line shows whether connection failures happened, and the second line shows whether execution failures happened.

Commands for Replica Management

The set of commands below provide functionality for the management of replicas, in terms of a global storage index, as well as a local replica lookup system.

In order to have support for replica management, the entries in the table need to have support for GUIDs. This is enabled, by using the makedir command instead of the createdir command, which takes exactly the same options, execpt, that permissions and the guid feature are switched on by default.

GUIDs are inserted as root via

Query> addentry /fdir/entry GUID beda818d-1090-489c-b9f9-6f3156a81828

If you want to play with this, use the uuidgen UNIX-command to create new guids. To list the guid values, use the -g switch in ls or stat.

The following commands allow the management of replicas of files, similar to a file catalogue local to a site. This functionality provies the possiblity to map GUIDs to SURLs (also known as PFNs, Physical File Names). Note that AMGA only supports one SURL per guid. This means that if you want to support multiple replicas (e.g. on multiple Storage Elements), you need to have one AMGA catalogue per SE:

• replica_add guid surl <expires>: Adds aSURL as a replica to the guid. That GUID must exist. Optionally it is possible to define an expiration time (e.g. for volatile storage). That time must be a date parseable by the database (this functionality is not implemented). The guid must already be known to the system, e.g. by adding an entry in a guid-enabled table with addentry. This is a root-only command.

• replica_remove surl [guid]: Removes a replica SURL of all files. It is possible to restrict the operation to guid only. This command requires root privileges as the permissions on the entry itself are not checkd.

• replica_get [guid ... | -l lfn ...l]: Returns the SURLs of all guids or lfns given.

• lfn_lookup guid looks up the LFN and the SURL of the guid.

The next set of commands allows to assign sites as locations of replicas to guids. This is usefull if the catalogue is being used in a global mode. In that case it can easily provide information on sites with replicas of a given file. This functionality is also referred to as a SiteIndex (SI) and AMGA stores the information in a highly-optimized bitfield. The Site Index functionality can be combined with the AMGA functionality for local replica management above. Note that the ideal mode of operation would be if local AMGA catalogues store SURLs and can be locally updated, replicating the information back to a global SI. However, this mode of operation is currently not supported by AMGA's replication functionality.

The SI functionality is highly optimzed in AMGA for the usage with Oracle, but also PostgreSQL was successfully tested and the performance is very good. The guids are stored in binary format as well as the storage index bitmap, so the information is stored in the most compact format. Indexes are used for row-lookup.

• replica_register guid site: Registers a replica for guid at the given site. The site can be either the site id (number) as returned by site_list, or the name of the site. This command requires root privileges.
• replica_unregiseter guid site: Unregisters a replica guid from a site. The site can either be the numerical id, or its name. This command requires root privileges.
• replica_list [guid ... | -l lfn ...l]: Returns the sites which hold replicas of all guids or lfns given.

The function lfn_lookup guid is used to retrieve the LFN and SURL from a given guid, however in AMGA 2.0 it only returns the directory in which a file resides.

---