This section provides a detailed description of the export process and the format and function of mapping files

An Overview of the Export Process

When performing an export, the exporter needs to read data from BMC Atrium Discovery's datastore, restructure it so that it matches the schema of the system it is being exported to and export it to the remote system. If any errors occur during the export of the data then the exporter needs to decide how much of the data to roll back.
To accomplish this, the exporter goes through the following steps:

1. Determines which system to send the data to, and with which connection parameters. This information comes from the adapter configuration specified by the user.
2. Runs a connection test to the specified remote system.
3. Determines which data is to be exported to the remote system, and how that data is to be restructured during the export.
   This information comes from the mapping set specified by the user. Each mapping file in the mapping set describes a dataset in BMC Atrium Discovery's datastore. For each mapping file, the specified data is retrieved, restructured and exported.
4. After all the mapping files have been run, performs any final tasks (logs the export statistics, writes any required manifests) and closes the connection.

The most complicated part of the export process is the restructuring of data. Each record in the BMC Atrium Discovery dataset specified in the mapping file is converted by the exporter into a set of configuration items (CIs). One CI is conceptually similar to a record in a table. A set of CIs can be thought of as a set of records in various tables that are linked by foreign keys. For example, a set of CIs could contain one "host" CI, four "IPAddress" CIs and a "CPU" CI.

Each set of CIs has one "Main" CI. All of the others are sub CIs. In the above example, the "Host" would be the main CI, and the IPAddress and CPU items would be sub CIs.

During the export, each record of the BMC Atrium Discovery data set is converted into one set of CIs. Each set of CIs is exported together. If the export of any CI in the set fails, then the whole set is rolled back. For example, if the CPU CI in our example cannot be inserted because a required field is left blank, then the Host and IPAddress items will not be inserted either.

An overview of the process for retrieving, restructuring and exporting the data for each mapping file is shown in the figure below.

Key Steps

There are four key steps in the process:

1. The query string is run. You can find more information about the Query section in a mapping file in The Mapping File Format.
2. The result of running that query is produced.
3. The records are transformed into a set of CIs. This is described in more detail in Transforming a BMC Atrium Discovery Dataset using a Mapping File.
4. The set of CIs are inserted into the database. This is described in more detail in Inserting CIs.

The Mapping File Format

This section provides an introduction to the mapping file format. It corresponds to step 1 in the Key Steps diagram above. For further details about the sections of the mapping files see A Closer look at Mapping Files.
Each mapping file is made up of two sections, the query section and the transformation section.

• The query is a standard search service query interpreted by BMC Atrium Discovery. BMC Atrium Discovery uses the query to retrieve information from the datastore and then returns the result to the exporter. Further information about search service can be found in the Search and Reporting Service.
• The transformation section specifies how the results of the query will be transformed into the appropriate format for publication by an adapter, for example, CSV files.

The following diagram shows the Query and Transform sections of a default mapping file. The diagram also shows the way the transformation section is divided into Main and Sub CIs.

Atrium Adapter Only

The default Atrium namespace is BMC.CORE. The exporter assumes that any BMC class that you specify in the cmdb-name attribute is in this namespace unless you explicitly specify a different namespace with the class name.

You do this by specifying the cmdb-name attribute as:
[namespace]:[classname]

For example, the class "MyClass" in the namespace "MyCo.Namespace" would be specified as:
cmdb-name="MyCo.Namespace:MyClass"

Transforming a BMC Atrium Discovery Dataset using a Mapping File

This section describes step 3 in the Key Steps diagram above.

A mapping file contains a BMC Atrium Discovery datastore query. When the mapping file is run, the query is executed on the datastore and the query result from this is used as the source data to the transformation specified in the mapping file.
Consider the following query:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as host_hostname,
      #RunningSoftware:HostedSoftware:Host:Host.name as host_name

This query returns the following result set:

The first two fields (Name and Description) have returned one value per record. The next two fields, on the other hand, are the result of traversals over a relationship (known as "key expressions" in the search service).

They each return a sequence of values: one value per relationship that was traversed. They are the result of traversing all relationships of the type RunningSoftware:HostedSoftware:Host from the BusinessApplicationInstance (BAI).

The first BAI (Payroll) had three such relationships, and so the host_hostname and host_name fields returned three values each for that BAI's record. The second BAI (Website) had four such relationships, while the last BAI (Employee Expenses) had two.

Note how both of the fields that returned sequences (host_hostname and host_name) returned sequences that correspond. The first entry in the Payroll's host_hostname field (webserv01) corresponds to the first entry in Payroll's host_name field. The second and third entries in each field also match.

Using these corresponding sequences, we can compile a list of Hosts that are related to each application. In our example, the Payroll application could be described as follows:

Name: Payroll
Description: The payroll application
  Hosts:
  Host 1
  Hostname: webserv01
  Name: webserv01.mycompany.com

  Host 2
  Hostname: London_orcl
  Name: london_orcl.mycompany.com

  Host 3
  Hostname: sap_01
  Name: sap_01.mycompany.com

We have taken one record from the result set and pivoted it, generating a BusinessApplicationInstance CI and three Host CIs from the record. This is how the transformation process works. Consider the following CI declarations from a mapping file (this is described in more detail in A Closer look at Mapping Files).

<ci cmdb-name="bai" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="Description"/>
  </ci>

<ci cmdb-name="host" collection="true">
    <field src="host_hostname" dest="HostHostName" identity="true"/>
    <field src="host_name" dest="HostName" identity="true"/>
    <relationship cmdb-name="hostedsoftware" direction="main-to-sub"/>
  </ci>

The first CI (the one declared "main") is the principal CI that this mapping file is concerned with. It is typically the node from which the various traversals start.

The sub-CI ("host") is generated from other fields in the result set. If its fields return sequences then you will need to set "collection='true'"; if you only expect one value per field then you can leave that declaration out.

The "relationship" element in the sub-CI tells the exporter how your main CI and sub CI are related. It is used when exporting to systems where the relationship has a name, such as Atrium. For the simpler adapters (such as CSV and RDB) it is ignored. If you intend to use the mapping file for these adapters only, you still need to specify the relationship, its name and direction but you can specify any values.

Inserting CIs

This section describes step 4 in the Key Steps diagram above.

In order for the Exporter to validate mapping files, one of the following must be specified for sub CIs (even if the mapping is intended for the CSV or RDB adapter):

• One field element with the attribute "identity='true'"
• A "node-reference-field='some-value'" attribute on the CI

For the RDB and CSV adapters the values will be ignored, but they still need to be specified.

When a CI is inserted, two pre-conditions are checked. The exporter queries the remote system based on the CI's remote identifier (if it is available) and then the CI's identity fields to determine if the CI is already present in the remote system. For further information about Remote Identifiers refer to The Remote Identifier - Atrium Adapter only.

• If the CI is present then it is inserted.
• If not present, it is created and the remote system's identifier for it is stored in BMC Atrium Discovery's datastore if a node reference field is specified for this CI.

The following flowchart shows the steps for inserting one CI.

In Atrium, all items (including relationships) MUST have at least their "Name" field populated.

The Relationship Between CIs

The main CI is related to all the others in the set (the sub CIs). Some sub-CIs can be a collection: for example, a Host CI can have multiple IPAddress CIs.

Relationships between the main and sub CIs are directional

• From the sub CI to the main CI (sub to main)
• From the main CI to the sub CI (main to sub)

Each relationship has a type

In Atrium, the BMC_HostedSystemComponents relationship points to the component, So a host is related to its hardware items, not vice versa.

The relationship is specified with the "relationship" sub-element of a Sub CI. The relationship element is required in a mapping file but is ignored by the CSV and RDB adapters.

Inserting a Set of CIs

One record from the result set becomes one set of main and sub CIs. A set of CIs is inserted in the following order:

1. All CIs that the main CI depends on (i.e. that are related to the main CI by sub to main relationships)
2. The main CI
3. All CIs that depend on the main CI (i.e. are related to the main CI by main to sub relationships)

A Closer look at Mapping Files

Query Section and the use of Timestamp

A sample of the Query section of the mapping file is shown below:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description, # as noderef,
      #RunningSoftware:HostedSoftware:Host:Host.# as host_noderef,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as hostname,
      #RunningSoftware:HostedSoftware:Host:Host.name as hosts_name

The Query section is built up of search service functions. For more information on how to build search queries please see the Search and Reporting Service.

The following search service functions are not supported by BMC Atrium Discovery Export: dq dq_band dq_metric

In the query section, the exporter makes a variable available that contains the time at which the exporter was last run. This variable is called "lastExportFinished" and is used with the function parseTime as follows:

parseTime("${lastExportFinished}")

This generates a timestamp that the datastore can recognize.

When this variable is encountered, the exporter substitutes the variable with the date that it was last run. The exporter then sends the search query to the datastore.

By unchecking the "Export changed items only" checkbox the exporter will set the LastExportFinished to 1 Jan 1980. This will result in a full export.

Example: Using the variable as part of a Where clause
This variable can be used as part of a Where clause.The following example will return items that have changed since the last time this exporter was run:

search Host
where parseTime("${lastExportFinished}") < modified(#)
show hostname

This variable can also be used with search services functions inside mapping file queries. For example, it can be used to filter on changes in dependencies between BAI and software collection.

Transformation Section

The transformation section is made up of a number of CIs. Each CI has a name (cmdb-name) and a number of field elements. There is one main CI and zero or more sub CIs. There can only be one main element (it has the attribute "main" set to true).

Main CI Transformation

In this section of the mapping file, the main attribute is set to "true", indicating that this is the main CI.

<ci cmdb-name="BMC_Application" main="true" 
node-reference-field="noderef">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="shortDescription"/>
</ci>

The name of the CI on the remote computer is BMC_Application.

The node-reference-field="noderef" – this is the datastore item reference for the exporter to write the remote system ID to.
The set of fields with identity = 'true' together uniquely identify this CI. Identity tags can be set on one or more fields.

If more than one field has "identity='true'" set then the exporter will only overwrite an existing item if it has identical values in all of the identity fields. In other words, multiple identity fields cause an AND operation, not an OR.

Sub CI Transformation

The direction of a relationship between the main and a sub CI can be one of the following:

• main-to-sub
• sub-to-main
  
  

The Remote Identifier - Atrium Adapter only

If the Atrium class that you're exporting to corresponds exactly to one BMC Atrium Discovery taxonomy kind (for example, an Atrium BMC_ComputerSystem corresponds to a BMC Atrium Discovery Host), then you can configure the mapping file so that Atrium's ID for the host is stored with the host in BMC Atrium Discovery's datastore.

This ID is then used in future exports to correctly reconcile items that are being exported with items that are already in Atrium. When used like this, the exporter can correctly reconcile items between BMC Atrium Discovery and Atrium, even when the item's identity fields have changed in either of the systems. The ID that is stored with the BMC Atrium Discovery node is referred to as the "remote ID".

This configuration is achieved by querying the BMC Atrium Discovery node's node handle from the datastore and associating it with the CI or sub-CI in the mapping file. Do the following:

• Query the node from the datastore by querying the field "#node". For example:

  search Host show name, host_name, #node as host_node_handle

• Specify which query result field contains the node handle in the mapping for the CI. For example:

  <ci cmdb-name="BMC_ComputerSystem" collection="true"
  node-reference-field="host_node_handle">

  For every host the exporter exports, it will store the Atrium ID as an attribute on the node in the BMC Atrium Discovery datastore.

Multiple adapter configurations could be exporting the same data. In order to prevent one adapter configuration's remote ids overwriting another's, the remote ids are stored per adapter configuration. Thus, if one Host node is exported through three adapter configurations, then that node will have three remote id attributes.

Parent-is-Identifier - Atrium adapter only

Some CIs can only be uniquely identified by their identity fields AND their relationship to another CI. For example, two computer systems may have the same model of CPU. Each individual CPU item can only be identified by the model name and the PC containing the CPU.
If parent-is-identifier="true" is set on the sub-CI then the exporter will only overwrite an item that has matching values in its identity fields AND is related to the same main CI.

Exporting deleted items - Atrium adapter only

Items in the BMC Atrium Discovery datastore can be deleted either automatically (e.g. by being aged out) or manually. If that item has been previously exported to the Atrium CMDB then the exporter can automatically delete it from the CMDB when the export is next run after the item being deleted in BMC Atrium Discovery.

You can do this by setting the delete-kind attribute on the mapping XML element:
{{<mapping description="this demonstrates how to delete items" delete-kind="Host"> }}
The delete-kind value has to be one of the types defined in the BMC Atrium Discovery taxonomy, for example, Host, SoftwareInstance, Cluster and you can only have one delete-kind attribute on a mapping element.

Errors during the mapping validation phase

Errors may be raised during the mapping validation phase. The following table describes these possible errors.