As the Tideway Foundation data store runs, many database entries are created and deleted. As this takes place, the database structure can become fragmented, meaning that there are 'gaps' in the data. The database files therefore tend to grow over time, even if the amount of useful data remains constant.

The solution to this situation is to periodically compact the data store. The compaction procedure defragments the databases, reclaiming the space wasted in the 'gaps'.

Back up appliance before compacting the data store You should always back up the data store by performing an Appliance Snapshot before compacting the data store. Failing to do so could result in data loss.

There are two different compaction tools available. The simplest and most common approach is to perform an online compaction. In some circumstances, it is useful to perform an offline copying compaction.

Online compaction

Online compaction is triggered using the tw_ds_online_compact command line tool. The command line options for the tool are:

Running compaction

Since it performs an online compaction, the tideway service must be running for the script to operate.

Running tw_ds_online_compact with no options triggers compaction of the whole data store. It prints out each node / relationship kind as it goes.

Since compaction touches all of the data in the databases, it is best if the data store is relatively quiet during the compaction process. By default, the tool therefore refuses to run if Discovery is running. Discovery should be stopped (either through the web user interface, or by using the tw_injectip script). Alternatively, you can force compaction to run regardless of the Discovery status by specifying the --force command line option.

Compaction of a particular node or relationship kind cannot be stopped once it has started. If you interrupt the script with ctrl-c, it will wait until it has finished the node / relationship kind it is working on before exiting.

Compacting individual node / relationship kinds

Rather than compacting the whole data store, you can compact individual node or relationship kinds by specifying them on the command line with --nodekind or --relkind respectively. The --list command line option outputs a list of all the node and relationship kinds stored in the data store.

Copying compaction

Instead of the usual online method, the data store can be compacted with an offline copying compaction that writes a new copy of the database files, performed by the tw_ds_compact script. This older compaction method is mainly supported for backwards compatibility, but it also has the ability to transform the data in two ways:

1. In Foundation 7.3, the way many data items are encoded has been improved, so they are now stored in a more efficient manner. When a pre-7.3 data store is used with 7.3, new data values that are written are written with the new encoding, but existing data values are not automatically encoded. The copying compaction script can recode data values to be stored in the more efficient 7.3 encoding.
2. The copying compaction can optionally skip any nodes and relationships that are marked as destroyed. That of course further reduces the size of the databases, but means some potentially important information is lost. In particular, superseded inferences are represented as destroyed relationships, so removing all the destroyed nodes / relationships causes the loss of some provenance information.

Preparing to Compact the Data Store

You must prepare the database using the following procedure:

1. Stop the Tideway Foundation services. As the tideway user, enter the following commands:

   $ sudo /sbin/service tideway stop

2. Change to the data directory. Enter:

   $ cd /usr/tideway/var/tideway.db/data/datadir

3. Run the database recovery tool to ensure the databases are in a consistent state. Enter:

   $ /usr/tideway/BerkeleyDB.4.7/bin/db_recover -v

4. Make a backup copy of all data files in /usr/tideway/var/tideway.db/data/datadir and transaction logs in /usr/tideway/var/tideway.db/logs
5. Remove the transaction region files. Enter:

   $ rm /usr/tideway/var/tideway.db/data/datadir/__db.*

6. Delete all files from the database logs directory. Enter:

   $ rm /usr/tideway/var/tideway.db/logs/*

The databases are now ready for compaction.

Compacting the Data Store

The tw_ds_compact script compacts the data store by copying the data files. The command takes the form

tw_ds_compact [options] <source directory> <destination directory>

The command line options for the tw_ds_compact script are described in the Table below.

The script accesses the database files directly, outside the usual transactional environment.

When you run the script, you must choose a destination directory for the new database files. Ideally you should store the new databases on a different disk, to minimise thrashing between reading the old files and writing the new ones. The new databases will generally be smaller than the originals, but you should ensure that there is at least the same amount of space as there is taken up by the current databases.

In the following example, the new database files are stored on /mnt/bigdisk.

1. Run the tw_ds_compact script:

   $ tw_ds_compact /usr/tideway/var/tideway.db/data/datadir /mnt/bigdisk

   
   The script outputs numbers to show its progress as it copies the databases. Redirect stderr to stdout using 2>&1. You may wish to record the information, in which case you can use tee to display the output and record it to a file:

   $ tw_ds_compact /usr/tideway/var/tideway.db/data/datadir /mnt/bigdisk 2>&1 | tee compact.log

2. When the script finishes, the original database files will be unmodified, and a new copy will exist in the destination directory.
3. To finish the operation, remove (or move) the old database files, and move the new files into the /usr/tideway/var/tideway.db/data/datadir directory.

   $ mv /usr/tideway/var/tideway.db/data/datadir/* /tmp/db/
   $ mv /mnt/bigdisk/* /usr/tideway/var/tideway.db/data/datadir

4. Finally, start the tideway services. Enter:

   $ sudo /sbin/service tideway start 

5. The transactional environment is rebuilt and the __db.* files return. A log.0000000001 file is created in the logs directory.
   The compaction is complete and the new database files are operational.