How to re-index existing indexes

Re-indexing is required, if you want to be able to search for event data of the past like payload, stacktrace, activity duration, etc. If you just want to search for process executions, filtered by time or domain object, re-indexing is not required.

In contrast, if you come from Elasticsearch 2, you have to re-index the indexes from v2 to v5 before you can migrate to Elasticsearch 6. In this scenario re-indexing is required! The following instructions describe re-indexation from v5 to v6 as a sample for re-indexation from v2 to v4.

Initial situation after upgrading your Elasticsearch Cluster from 5 to 6 and upgraded nJAMS Server to 5.0:

When the upgrade is done and nJAMS Server is started again, you can enter the “Index Management” page to see the current state of your indexes of your Elasticsearch Cluster. Go to Administration > Connections > Indexer > Tab Index Management:

A:

REFRESH updates the current page to see the last changes to the indexes

HIDE non-nJAMS indexes only show indexes used by nJAMS

MAINTENANCE Mode enables you to re-index the indexes. As soon as you activate the Maintenance Mode, Indexer and Data Providers will stop. That means, when you are in Maintenance Mode, message processing is off, as well as searching for monitoring data.

DELETE all nJAMS indices removes all indexes related to nJAMS. Be careful, when you confirm the following confirmation prompt, all data will be lost. Please note, once all indexes have been removed, nJAMS Server will immediately start creating new indexes again.

B:

Index state let you know, whether there are indexes left of previous version in your Elasticsearch

C:

This is the list of indexes of your Elasticsearch Cluster used by nJAMS:

Name is the name of the index

Type indicates the index type, respectively the usage of the index

Expiration indicates the date, when the index will expire

Primary documents indicates the number of documents, which are relevant for re-indexing

Size indicates the size of the index

Estimated re-index duration is a rough estimation about how long a re-indexing of an index may take. Initially the estimated time is “unknown”. As soon as you re-indexed the first index, nJAMS Server can estimate roughly the time of the other indexes. Please note, re-indexing a data index is much more time consuming than re-indexing a statistics index, not only because of the amount of data, but because of the structure of the index. Since the duration will be estimated according to the index(es) you indexed, the estimated duration may lead to misleading results, depending on index type. For example, you indexed a data index first. Afterwards the remaining indexes are estimated according to that baseline. This approach results in overestimating the duration for re-indexing of statistic indexes, i.e. re-indexing the statistic indexes will be quicker than estimated. On the other hand, when you re-index a statistic index first, the duration of data indexes will be underestimated, i.e. re-indexing the data indexes will take more time than estimated. To be on the safe side, we recommend to re-index a data index first.

Index version is the version of the index format

Read only indicates the index is read only, no data can be inserted or updated in that index. When re-indexing a v5 index, a new index in format v6 is created. The source index in v5 will remain in read-only state.

Actions let you re-index or remove the selected index. Both actions are only available, when Maintenance Mode is active. Re-indexing is only available for supported index types. You can only delete nJAMS related indexes and “.scripts” indexes.

Procedure for re-indexing:

1. Enter Administration > Connections > Indexer > Tab Index Management

2. Activate Maintenance Mode

3. Select the index you want to get re-indexed

4. Click on the pen icon in column Actions, the following dialog opens:

   

   Source index name: name of the source index

   Target index name: new name for the re-indexed index. In most cases there is no need to change the proposed name.

   Delete source index: when this option is enabled, the source index will be removed, if re-indexing was successful. When you disable this option, the source index will remain in state read-only. Please note: if there are both indexes (source and target index) available, you will get duplicate results in nJAMS GUI. It is recommended to remove the source indexes immediately after re-indexing, so just leave this option enabled.

   Estimated duration: estimated period of time for re-indexing

   Remaining duration: time left

   Documents processed: number of document that have been processed during re-indexing

   Task-Id: this is the Id of the task in Elasticsearch.

   Result: expected to be successful, of course.

5. Start re-indexing

6. Wait for finishing re-indexing. Depending on type and size of the index and the performance of your Elasticsearch Cluster, re-indexing may take a while. You can close the dialog by clicking on ‘X’, or you can cancel the re-indexing process by clicking on ‘Cancel’:

When you cancel the re-indexing process, you can start re-indexing the index once again.

When you close the dialog, the re-indexing process will continue to run in background. But you cannot follow the process for this index anymore. It is recommended to keep the dialog open until re-indexing is finished.

Note

It is possible to close the re-indexing dialog during re-indexing and start re-indexing another index in parallel. Nevertheless, it is not recommended to proceed in this way! The Elasticsearch Cluster may get out of heap, which may cause damage to your data.

7. When re-indexing was successful, you can see the result here:

   

   There is now a new index in format v6 and the previous index has been removed.

8. Continue re-indexing with the next index.