MDS 2.4 New Features
The MDS 2.4 release includes fixes and enhancements for enhanced scalability, reliability, and stability.
MDS 2.4 contains the following new features:
- Flexible handling of stale data via a new parameter in the grid-info-slapd.conf file
- Improved GRIS/GIIS processing by allowing multiple GRISes in a slapd server
Note that these features are intended for advanced MDS2 system administrators and users. If you are new to MDS2, you can begin by reading the User's Guide.
These features are described in detail in the following sections:
Flexible Handling of Stale Data
When a GIIS receives a query, the following occurs:
For every registrant, the GIIS checks its cache and if the query can be satisfied from cache, responds from there. Otherwise the GIIS sends a query to that registrant.
Queries are sent to each registrant in parallel – it is not necessary for one query to complete before the next query is sent.
This fetching process for each registrant should be completed before the timeout value of the registrant expires. If a registrant’s response is not received before the timeout, it is not returned by the GIIS for the present query. But if the registrant’s response is received later, then this response is cached by the GIIS and used when it processes a later query, depending on the timeout value.
Consider the following diagram:
D1 D2
| |
----+--------+---+---+-----+-----+------->
| | | |
Q1 Q2 T1 T2
The above timing diagram shows a simple example of the stale data pickup
process. When a GRIS registers to a GIIS with a timeout value (Ti - Qi) of
10 seconds, but the GRIS responds to the query in 15 seconds (Di - Qi), the
first query (Q1) to the GIIS does not include the output of the GRIS because
the GRIS' data is not ready in 10 seconds. But a second query (Q2) to the
GIIS picks up the GRIS' data that was requested in the previous query (Q1)
to the GIIS.
allowstaledata Parameter
A new parameter in the grid-info-slapd.conf file, allowstaledata, gives you the choice of publishing stale data into MDS, or publishing data only when it is received on time.
The value of allowstaledata can be either yes or no. A yes value (the default) means that a grid-info-search query to a GIIS may include the output of a registrant GRIS that was requested in the previous query, but was not returned before the timeout value expired. A no value means that a grid-info-search query to a GIIS will include the output of a registrant GRIS only if the data requested by the current query is received on time.
The circumstances under which to publish only current data or include stale data depend on the characteristics of the client applications. If it is important to use only current/valid data in a client application, then setting allowstaledata to no is preferable.
Example
The following is an example of a section of a grid-info-slapd.conf file with allowstaledata set to no, so the GIIS/GRIS publishes data only if it is received on time:
database ldif
suffix "Mds-Vo-name=local, o=Grid"
conf /hosta/globus/install/etc/grid-info-resource-ldif.conf
anonymousbind yes
allowstaledata no
access to * by * write
database giis
suffix "Mds-Vo-name=site, o=Grid"
conf /hosta/globus/install/etc/grid-info-site-giis.conf
policyfile /hosta/globus/install/etc/grid-info-site-policy.conf
anonymousbind yes
allowstaledata no
access to * by * write
Note that the allowstaledata parameter can be set individually for each GIIS or GRIS, within the corresponding database definition. Global setting of allowstaledata is not possible.
Improved GRIS/GIIS Processing
The default MDS configuration allows for a single GRIS and one GIIS in a server. In this configuration, a GIIS queries its registrants concurrently, but each GRIS invokes its information providers sequentially. Sometimes this behavior can cause a long delay in GRIS response time, such as when there are a large number of information providers or several information providers that are very slow.
The default MDS configuration can be illustrated as shown in the following example:
Each information provider in a GRIS has its own cachetime value, so that its data can be cached for a certain time period. These cachetime values of the information providers vary according to the types of information providers. For example, the cachetime value of the operating system version information should be larger than the cachetime value of the CPU load information.
But the GIIS sets only one cachetime value for the GRIS when it first receives a registration message from the GRIS, and it caches all of the GRIS' data for the duration of that cachetime value. By default, the cachetime value of the GRIS is set to be the same as the smallest cachetime value of the GRIS’ information providers. Therefore, unnecessary network traffic and unnecessary data processing may occur.
Multiple GRISes in a slapd Server
To avoid the problems that can be caused by sequential invocation of information providers and a single cachetime value, MDS 2.4 allows multiple GRISes to be configured in a single slapd server. This allows information providers to be split across GRISes so that the providers can be invoked concurrently. A multiple GRIS configuration therefore reduces the amount of time required for the entire information provider invocation sequence.
The following diagram shows an example of an MDS configuration with multiple GRISes, where the information providers for each GRIS can be thought of as constituting a distinct group. See Determining the Number of grid-info-resource-ldif.conf Files below for specific examples of these kinds of groups.
Determining the Need for Multiple GRISes
The default information providers that come with MDS usually run in a short period of time (single digit seconds) on most of the machines. Therefore, multiple GRISes are mostly needed when there are additional information providers. Multiple GRISes can also be used with the default information providers to eliminate unnecessary network traffic and data processing as described above.
Basically, you can use a single GRIS when all the information providers run quickly enough to meet the purpose of the client application. Otherwise, it is good to use multiple GRISes, especially when the delay is too long to ignore because of slow information providers or a large number of information providers.
Splitting the Configuration Into Multiple GRISes
If you determine that a multiple-GRIS configuration makes sense for your particular MDS environment, there are three configuration files you will need to modify:
- grid-info-resource-ldif.conf
- grid-info-slapd.conf
- grid-info-resource-register.conf
These modifications are described in detail in the following sections.
Modifying grid-info-resource-ldif.conf
The grid-info-resource-ldif.conf file determines which GRIS information providers are active and available to send data to the GIIS resources to which a GRIS is registering. This file describes the core information providers as well as any custom providers for the GRIS.
To create a multiple-GRIS configuration, you need split the default grid-info-resource-ldif.conf file into multiple files (one for each group of information providers). The following section describes how to determine the number of grid-info-resource-ldif.conf files you will need.
Determining the Number of grid-info-resource-ldif.conf Files
Consider the dn value, response time, and cachetime of each information provider to decide on the number of grid-info-resource-ldif.conf files required for your new configuration. Use the following procedure:
- Create initial information provider groups based on dn values.
- Determine the response time of each provider in the initial groups and then combine them into new groups accordingly.
- Optionally, subgroup each provider group based on cachetime values.
These steps are described in detail in the following sections.
Create Initial Information Provider Groups
Each information provider produces objects that have individual dn values. These individual dn values should be the same as the information provider's dn value or should be the children of the information provider's dn value.
First, group the information providers with the same dn values. These initial groups will then be combined into new groups according to the resulting response time of each group.
For example, if there are six information providers, IP1 - IP6, where the dn value of IP2 and the dn value of IP3 are the same, group these providers into five initial groups, IPG1 - IPG5, where IPG2 contains the two information providers with the same dn values.
Determine Provider Response Time and Combine Groups
The response time of each information provider can be obtained by running it as described in the grid-info-resource-ldif.conf file. For example, the following is an active object in the grid-info-resource-ldif.conf file:
# generate OS info
dn: Mds-Software-deployment=operating system, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-os-uname
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 21600 -keepto-secs 240800
cachetime: 10800
timelimit: 20
sizelimit: 1
The following command gives the response time of the information provider:
% time /hosta/globus/install/libexec/grid-info-os-uname \
-devclassobj -devobjs \
-dn Mds-Host-hn=hosta.isi.edu,Mds-Vo-name=local,o=grid \
-validto-secs 21600 -keepto-secs 240800
The output of this command is as follows:
dn: Mds-Software-deployment=operating system,
Mds-Host-hn=hosta.isi.edu,Mds-Vo-name=local,o=grid
objectclass: MdsSoftware
objectclass: MdsOs
Mds-Software-deployment: operating system
Mds-Os-name: Linux
Mds-Os-release: 2.4.7-10
Mds-Os-version: 1 Thu Sep 6 17:27:27 EDT 2001
0.020u 0.000s 0:01.03 66.6% 0+0k 0+0io 1115pf+0w
Therefore the above command takes 1.03 seconds to produce the object.
After running all the information providers, combine the initial groups into new groups so that the maximum value of the processing times of all the groups is under your target response time.
In the above example, if the response times of groups IPG1 - IPG5 are 5, 10, 1, 2, and 3 seconds respectively, then combine the groups into three new groups where the first includes IPG1, the second includes IPG2, and the third includes IPG3 - IPG5. The processing time of Group 1 is 5 seconds, the processing time of Group 2 is 10 seconds, and the processing time of Group 3 is 6 seconds. There is no need to subgroup Group 3 because the response time of all the information providers is determined by the slowest information provider group.
Having determined these three groups means that you will need three grid-info-resource-ldif.conf files, one for each group.
Note that you don't need to modify each information provider's dn value within a group. But you need to have a new group dn value that will be used later as suffix in the grid-info-slapd.conf file. The new dn value of the group should be the parent of all the dn values of the information providers in the group.
Subgroup Based on cachetime Values
The cachetime is not as critical as the response time in determining the required number of grid-info-resource-ldif.conf files. So it's optional to consider the cachetime of each information provider. To avoid unnecessary network traffic and data processing, you can subgroup each of the provider groups having the same cachetime values.
Naming the grid-info-resource-ldif.conf Files
The new grid-info-resource-ldif.conf files you create can be called anything you like. We suggest naming the files in a way that references the type of data produced by their associated information providers, for example, grid-info-resource-ldif-host.conf, grid-info-resource-ldif-cpu.conf, and grid-info-resource-ldif-system.conf. These names are used in the examples below.
So the previous diagram of a multiple-GRIS configuration would look like this for our specific example:
Example Files
Continuing on from our example, you determined that what best suits your configuration is to split the original grid-info-resource-ldif.conf file into the three new files named above. The original grid-info-resource-ldif.conf file is shown below, followed by the new files.
Original single grid-info-resource-ldif.conf file:
# this file contains the default GRIS providers and must be configured
# for a particular platform to specialize the template...
# generate top-level Mds-Host-hn=host object
dn: Mds-Host-hn=hosta.isi.edu, Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-platform-merged
args: -dn Mds-Host-hn=hosta.isi.edu,Mds-Vo-name=local,o=grid
-validto-secs 3600 -keepto-secs 240800
cachetime: 1800
timelimit: 20
sizelimit: 1
# generate CPU availablity information
dn: Mds-Device-Group-name=processors, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-cpufast-uptime
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 60 -keepto-secs 240800
cachetime: 30
timelimit: 20
sizelimit: 100
# generate CPU inventory (hidden cache) every 12 hours
dn: Mds-Device-Group-name=processors, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-cpu-linux
args: -noobjs
cachetime: 43200
timelimit: 20
sizelimit: 1
# generate OS info
dn: Mds-Software-deployment=operating system, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-os-uname
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 21600 -keepto-secs 240800
cachetime: 10800
timelimit: 20
sizelimit: 1
# generate memory info
dn: Mds-Device-Group-name=memory, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-mem-linux
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 3600 -keepto-secs 240800
cachetime: 1800
timelimit: 10
sizelimit: 3
# generate filesystem info
dn: Mds-Device-Group-name=filesystems, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-fs-posix
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 3600 -keepto-secs 21600
cachetime: 1800
timelimit: 20
sizelimit: 20
New grid-info-resource-ldif-host.conf file:
# generate top-level Mds-Host-hn=host object
dn: Mds-Host-hn=hosta.isi.edu, Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-platform-merged
args: -dn Mds-Host-hn=hosta.isi.edu,Mds-Vo-name=local,o=grid
-validto-secs 3600 -keepto-secs 240800
cachetime: 1800
timelimit: 20
sizelimit: 1
New grid-info-resource-ldif-cpu.conf file:
# generate CPU availablity information
dn: Mds-Device-Group-name=processors, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-cpufast-uptime
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 60 -keepto-secs 240800
cachetime: 30
timelimit: 20
sizelimit: 100
# generate CPU inventory (hidden cache) every 12 hours
dn: Mds-Device-Group-name=processors, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-cpu-linux
args: -noobjs
cachetime: 43200
timelimit: 20
sizelimit: 1
New grid-info-resource-ldif-system.conf file:
# generate OS info
dn: Mds-Software-deployment=operating system,
Mds-Host-hn=hosta.isi.edu, Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-os-uname
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 21600 -keepto-secs 240800
cachetime: 10800
timelimit: 20
sizelimit: 1
# generate memory info
dn: Mds-Device-Group-name=memory, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-mem-linux
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 3600 -keepto-secs 240800
cachetime: 1800
timelimit: 10
sizelimit: 3
# generate filesystem info
dn: Mds-Device-Group-name=filesystems, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
objectclass: GlobusTop
objectclass: GlobusActiveObject
objectclass: GlobusActiveSearch
type: exec
path: /hosta/globus/install/libexec
base: grid-info-fs-posix
args: -devclassobj -devobjs -dn Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local,o=grid -validto-secs 3600 -keepto-secs 21600
cachetime: 1800
timelimit: 20
sizelimit: 20
Modifying grid-info-slapd.conf
The grid-info-slapd.conf file needs to be modified so that it contains an ldif database definition for each new grid-info-resource-ldif.conf file that you create.
To continue with our example, the following is the original grid-info-slapd.conf file:
schemacheck off
# The number of worker threads in the thread pool.
threads 32
# The maximum number of entries (objects) to return from a search
# operation.
sizelimit 500
# The maximum number of seconds (in real time) slapd will spend
# answering a search request.
timelimit 3600
include /hosta/globus/install/etc/openldap/schema/core.schema
include /hosta/globus/install/etc/grid-info-resource.schema
pidfile /hosta/globus/install/var/resourceslapd.pid
argsfile /hosta/globus/install/var/resourceslapd.args
modulepath /hosta/globus/install/libexec/openldap/gcc32dbgpthr
moduleload libback_ldif.la
moduleload libback_giis.la
database ldif
suffix "Mds-Vo-name=local, o=Grid"
conf /hosta/globus/install/etc/grid-info-resource-ldif.conf
anonymousbind yes
allowstaledata yes
access to * by * write
database giis
suffix "Mds-Vo-name=site, o=Grid"
conf /hosta/globus/install/etc/grid-info-site-giis.conf
policyfile /hosta/globus/install/etc/grid-info-site-policy.conf
anonymousbind yes
allowstaledata yes
access to * by * write
This file should be modified to have three ldif database definitions to match the number of new grid-info-resource-ldif.conf files in our example.
The order of the ldif database definitions in the grid-info-slapd.conf file is very important because OpenLDAP selects the first matching database in the order. For ldif database definitions that have a parent/child relationship, OpenLDAP requires that the parent definitions be listed after all of the children in the database order.
Also make sure that the last ldif database definition in the order is Mds-Vo-name=local, o=Grid, because only the GRIS with this suffix produces the GlobusStub object that is used to create internal objects in the object tree.
The values of the conf attributes in the ldif database definitions should include the names of the new grid-info-resource-ldif.conf files.
The value of the threads attribute should be greater than the number of GRISes in the configuration. Otherwise, the information providers might not be invoked concurrently.
The following is the new grid-info-slapd.conf file, modified to reflect the changes in our example:
schemacheck off
# The number of worker threads in the thread pool.
threads 32
# The maximum number of entries (objects) to return from a search
# operation.
sizelimit 500
# The maximum number of seconds (in real time) slapd will spend answering a
# search request.
timelimit 3600
include /hosta/globus/install/etc/openldap/schema/core.schema
include /hosta/globus/install/etc/grid-info-resource.schema
pidfile /hosta/globus/install/var/resourceslapd.pid
argsfile /hosta/globus/install/var/resourceslapd.args
modulepath /hosta/globus/install/libexec/openldap/gcc32dbgpthr
moduleload libback_ldif.la
moduleload libback_giis.la
database ldif
suffix "Mds-Device-Group-name=processors, Mds-Host-hn=hosta.isi.edu, Mds-Vo-name=local, o=grid"
conf /hosta/globus/install/etc/grid-info-resource-ldif-cpu.conf
anonymousbind yes
allowstaledata yes
access to * by * write
database ldif
suffix "Mds-Host-hn=hosta.isi.edu, Mds-Vo-name=local, o=grid"
conf /hosta/globus/install/etc/grid-info-resource-ldif-host.conf
anonymousbind yes
allowstaledata yes
access to * by * write
database ldif
suffix "Mds-Vo-name=local, o=Grid"
conf /hosta/globus/install/etc/grid-info-resource-ldif-system.conf
anonymousbind yes
allowstaledata yes
access to * by * write
database giis
suffix "Mds-Vo-name=site, o=Grid"
conf /hosta/globus/install/etc/grid-info-site-giis.conf
policyfile /hosta/globus/install/etc/grid-info-site-policy.conf
anonymousbind yes
allowstaledata yes
access to * by * write
Modifying grid-info-resource-register.conf
The grid-info-resource-register.conf file needs to be modified so that it contains one registration object for each new GRIS that you create.
To continue with our example, the following is the original grid-info-resource-register.conf file:
# for default MDS 2.4 install
# register this server GRIS to this server GIIS
dn: Mds-Vo-Op-name=register, Mds-Vo-name=site, o=grid
regtype: mdsreg2
reghn: hosta.isi.edu
regport: 2135
regperiod: 600
type: ldap
hn: hosta.isi.edu
port: 2135
rootdn: Mds-Vo-name=local, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 30
bindmethod: ANONYM-ONLY
This file should be modified to have three registration objects to match the number of new GRISes in our example. For clarity, these registration objects can be identified via comment lines in the file.
The values of rootdn in the grid-info-resource-register.conf file should match the values of suffixes in the grid-info-slapd.conf file.
The values of cachettl in each of the registration objects should match the smallest cachetime values in each of the corresponding new grid-info-resource-ldif.conf files, since there are now multiple cachetime values replacing the original single cachetime value in the original grid-info-resource-ldif.conf file.
The following is the new grid-info-resource-register.conf file, modified to reflect the changes in our example:
# for custom MDS 2.4 install
# register this CPU info provider GRIS to this server GIIS
dn: Mds-Vo-Op-name=register, Mds-Vo-name=site, o=grid
regtype: mdsreg2
reghn: hosta.isi.edu
regport: 2135
regperiod: 600
type: ldap
hn: hosta.isi.edu
port: 2135
rootdn: Mds-Device-Group-name=processors, Mds-Host-hn=hosta.isi.edu,
Mds-Vo-name=local, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 30
bindmethod: ANONYM-ONLY
# for custom MDS 2.4 install
# register this host info provider GRIS to this server GIIS
dn: Mds-Vo-Op-name=register, Mds-Vo-name=site, o=grid
regtype: mdsreg2
reghn: hosta.isi.edu
regport: 2135
regperiod: 600
type: ldap
hn: hosta.isi.edu
port: 2135
rootdn: Mds-Host-hn=hosta.isi.edu, Mds-Vo-name=local, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 1800
bindmethod: ANONYM-ONLY
# for custom MDS 2.4 install
# register this system info provider GRIS to this server GIIS
dn: Mds-Vo-Op-name=register, Mds-Vo-name=site, o=grid
regtype: mdsreg2
reghn: hosta.isi.edu
regport: 2135
regperiod: 600
type: ldap
hn: hosta.isi.edu
port: 2135
rootdn: Mds-Vo-name=local, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 1800
bindmethod: ANONYM-ONLY