This section provides a detailed description of the export process and the format and function of mapping files

Export process overview

When performing an export, the exporter needs to read data from BMC Atrium Discovery's datastore, restructure it so that it matches the schema of the system it is being exported to and export it to the remote system. If any errors occur during the export of the data then the exporter needs to decide how much of the data to roll back.
To accomplish this, the exporter goes through the following steps:

1. Determines which system to send the data to, and with which connection parameters. This information comes from the adapter configuration specified by the user.
2. Runs a connection test to the specified remote system.
3. Determines which data is to be exported to the remote system, and how that data is to be restructured during the export.
   This information comes from the mapping set specified by the user. Each mapping file in the mapping set describes a dataset in BMC Atrium Discovery's datastore. For each mapping file, the specified data is retrieved, restructured and exported.
4. After all the mapping files have been run, performs any final tasks (logs the export statistics, writes any required manifests) and closes the connection.

The most complicated part of the export process is the restructuring of data. Each record in the BMC Atrium Discovery dataset specified in the mapping file is converted by the exporter into a set of configuration items (CIs). One CI is conceptually similar to a record in a table. A set of CIs can be thought of as a set of records in various tables that are linked by foreign keys. For example, a set of CIs could contain one "host" CI, four "IPAddress" CIs and a "CPU" CI.

Each set of CIs has one "Main" CI. All of the others are sub CIs. In the above example, the "Host" would be the main CI, and the IPAddress and CPU items would be sub CIs.

During the export, each record of the BMC Atrium Discovery data set is converted into one set of CIs. Each set of CIs is exported together. If the export of any CI in the set fails, then the whole set is rolled back. For example, if the CPU CI in our example cannot be inserted because a required field is left blank, then the Host and IPAddress items will not be inserted either.

An overview of the process for retrieving, restructuring and exporting the data for each mapping file is shown in the figure below.

Key Steps

There are four key steps in the process:

1. The query string is run. You can find more information about the Query section in a mapping file in The Mapping File Format.
2. The result of running that query is produced.
3. The records are transformed into a set of CIs. This is described in more detail in Transforming a BMC Atrium Discovery Dataset using a Mapping File.
4. The set of CIs are inserted into the database.

The Mapping File Format

This section provides an introduction to the mapping file format. It corresponds to step 1 in the Key Steps diagram above. For further details about the sections of the mapping files see A Closer look at Mapping Files.
Each mapping file is made up of two sections, the query section and the transformation section.

• The query is a standard search service query interpreted by BMC Atrium Discovery. BMC Atrium Discovery uses the query to retrieve information from the datastore and then returns the result to the exporter. Further information about search service can be found in the Search and Reporting Service.
• The transformation section specifies how the results of the query will be transformed into the appropriate format for publication by an adapter, for example, CSV files.

The following diagram shows the Query and Transform sections of a default mapping file. The diagram also shows the way the transformation section is divided into Main and Sub CIs.

Transforming a BMC Atrium Discovery Dataset using a Mapping File

This section describes step 3 in the Key Steps diagram above.

A mapping file contains a BMC Atrium Discovery datastore query. When the mapping file is run, the query is executed on the datastore and the query result from this is used as the source data to the transformation specified in the mapping file.
Consider the following query:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as host_hostname,
      #RunningSoftware:HostedSoftware:Host:Host.local_fqdn as host_fqdn

This query returns the following result set:

The first two fields (Name and Description) have returned one value per record. The next two fields, on the other hand, are the result of key expression traversals over a relationship. They each return a sequence of values: one value per relationship that was traversed. They are the result of traversing all relationships of the type RunningSoftware:HostedSoftware:Host from the BusinessApplicationInstance (BAI). (Note that there is no need to use explode to cause the key expressions to be treated as separate rows in the output.)

The first BAI (Payroll) had three such relationships, and so the host_hostname and host_fqdn fields returned three values each for that BAI's record. The second BAI (Website) had four such relationships, while the last BAI (Employee Expenses) had two.

Note how both of the fields that returned sequences (host_hostname and host_fqdn) returned sequences that correspond. The first entry in the Payroll's host_hostname field (webserv01) corresponds to the first entry in Payroll's host_fqdn field. The second and third entries in each field also match.

Using these corresponding sequences, we can compile a list of Hosts that are related to each application. In our example, the Payroll application could be described as follows:

Name: Payroll
Description: The payroll application
  Hosts:
  Host 1
  Hostname: webserv01
  FQDN: webserv01.mycompany.com

  Host 2
  Hostname: London_orcl
  FQDN: london_orcl.mycompany.com

  Host 3
  Hostname: sap_01
  FQDN: sap_01.mycompany.com

We have taken one record from the result set and pivoted it, generating a BusinessApplicationInstance CI and three Host CIs from the record. This is how the transformation process works. Consider the following CI declarations from a mapping file (this is described in more detail in A Closer look at Mapping Files).

<ci cmdb-name="bai" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="Description"/>
  </ci>

<ci cmdb-name="host" collection="true">
    <field src="host_hostname" dest="HostHostName" identity="true"/>
    <field src="host_fqdn" dest="HostFQDN" identity="true"/>
    <relationship cmdb-name="hostedsoftware" direction="main-to-sub"/>
  </ci>

The first CI (the one declared "main") is the principal CI that this mapping file is concerned with. It is typically the node from which the various traversals start.

The sub-CI ("host") is generated from other fields in the result set. If its fields return sequences then you will need to set "collection='true'"; if you only expect one value per field then you can leave that declaration out.

The "relationship" element in the sub-CI tells the exporter how your main CI and sub CI are related. It is used when exporting to systems where the relationship has a name, such as Atrium CMDB. For the simpler adapters (such as CSV and RDB) it is ignored. If you intend to use the mapping file for these adapters only, you still need to specify the relationship, its name and direction but you can specify any values.

In order for the Exporter to validate mapping files, at least one field in each CI must be given the attribute "identity='true'".

A Closer look at Mapping Files

Query Section and the use of Timestamp

A sample of the Query section of the mapping file is shown below:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description, # as noderef,
      #RunningSoftware:HostedSoftware:Host:Host.# as host_noderef,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as hostname,
      #RunningSoftware:HostedSoftware:Host:Host.name as hosts_name

The Query section is built up of search service functions. For more information on how to build search queries please see the Search and Reporting Service.

The following search service functions are not supported by BMC Atrium Discovery Export: dq dq_band dq_metric

In the query section, the exporter makes a variable available that contains the time at which the exporter was last run. This variable is called "lastExportFinished" and is used with the function parseTime as follows:

parseTime("${lastExportFinished}")

This generates a timestamp that the datastore can recognize.

When this variable is encountered, the exporter substitutes the variable with the date that it was last run. The exporter then sends the search query to the datastore.

By unchecking the "Export changed items only" checkbox the exporter will set the lastExportFinished to 1 Jan 1980. This will result in a full export.

Example: Using the variable as part of a where clause
This variable can be used as part of a where clause.The following example will return items that have changed since the last time this exporter was run:

search Host
where parseTime("${lastExportFinished}") < modified(#)
show hostname

This variable can also be used with search services functions inside mapping file queries. For example, it can be used to filter on changes in dependencies between BAI and software collection.

If you have other conditions to place in the query's where clause, it is generally best to put the other conditions before the modified check, to avoid comparing modified times of many nodes that do not match the condition. e.g.

search Host
where os_type = "Windows" and
parseTime("${lastExportFinished}") < modified(#)
show hostname

Transformation Section

The transformation section is made up of a number of CIs. Each CI has a name (cmdb-name) and a number of field elements. There is one main CI and zero or more sub CIs. There can only be one main element (it has the attribute "main" set to true).

Main CI Transformation

In this section of the mapping file, the main attribute is set to "true", indicating that this is the main CI.

<ci cmdb-name="BMC_Application" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="shortDescription"/>
</ci>

The name of the CI on the remote computer is BMC_Application.

The set of fields with identity = 'true' together uniquely identify this CI. Identity tags can be set on one or more fields.

If more than one field has "identity='true'" set then the exporter will only overwrite an existing item if it has identical values in all of the identity fields. In other words, multiple identity fields cause an AND operation, not an OR.

Errors during the mapping validation phase

Errors may be raised during the mapping validation phase. The following table describes these possible errors.