Connections

The category Connections allows nJAMS Administrators to configure connections to Indexer and various JMS providers. Furthermore, the nJAMS Administrator can optionally create connections to a directory service (LDAP) and a mail server (SMTP).

For proper operation of nJAMS Server the Indexer and at least one JMS connection must be configured. Mail server configuration is optional; however, if users should be notified by the system automatically, or be able to reset their (forgotton) passwords using the nJAMS GUI, the SMTP connection must be configured. LDAP configuration is optional too and allows you to import users and roles from a directory service.

Indexer

The Indexer component is responsible for communication with Elasticsearch. There are mainly two tasks that are performed by the Indexer component:

1. Storing log messages into Elasticsearch and indexing log messages for fast retrieval.

2. Retrieving data from Elasticsearch and providing to nJAMS GUI.

The Indexer page allows nJAMS Administrators to assign an Elasticsearch cluster to nJAMS Server and to start / stop the Indexer component. The following chapters explain how to configure and maintain the Indexer.

Configuration:

This is the Indexer configuration and administration page:

1. The following indicators of the Elasticsearch cluster state are provided:

   

   Setting	Description
   Cluster Health State	State of the Elasticsearch cluster can be: green: cluster is operational yellow: cluster is disturbed, but operational without data loss red: cluster is inoperable, Message Processing is stopped
   Client On/Off	Turns Indexer Client on or off. It is required to stop the Client before its configuration can be modified. And it is also necessary to Stop the Client for maintenance work at the Elasticsearch cluster, e.g. creating snapshots from cluster, etc. Note When you stop the Indexer, be aware that Message Processing is stopped as well as searching and data retrieval in nJAMS GUI.
   Client Status	Status of Indexer Client (Indexer): CONNECTED: Client is fully connected and working WRITE-BLOCKED: Client is in read-only mode and does not write log messages into Elasticsearch STOPPING: Client is about to shut down connection to Elasticsearch cluster STOPPED: Client is stopped. There is no connection to Elasticsearch.
   Client Mode	Indicates whether the Indexer Client is used with any restrictions.
   Total Nodes	Number of total nodes of Elasticsearch cluster
   Data Nodes	Number of total Data nodes of Elasticsearch cluster
   Indices	Total number of Indexes
   Active Shards	Number of Active Shards
   Primary Shards	Number of Primary Shards
   Unassigned Shards	Number of Unassigned Shards
   Initializing Shards	Number of Initializing Shards
   Relocating Shards	Number of Relocating Shards
   Documents (L/T)	Number of documents: L: Number of documents followed from log messages represents the number of log entries in nJAMS Server. T: Total number of documents, includes all kind of data.
   Deleted Documents	Number of documents that have been deleted according to data maintenance settings
   Data Size	This is the actual size of the Cluster storage
   Pending Retries	Total numbver of requests pending for rescheduling
   Stored Failures	Total number of failed requests stored in database

2. The connection parameters of the Indexer component are as follows:

   

   Setting	Description
   Autostart	Enables Autostart option of Indexer component during start of nJAMS Server application
   Cluster name	Name of the Elasticsearch cluster
   Client name	Optional: Client node name that represents nJAMS Server by name in Elastichsearch Cluster
   Management URL	Optional: URL to an Elasticsearch Management Tool, e.g. ‘Kopf’. This URL will be used when you click on ‘OPEN Management Tool’.
   Enable sniffing	By default, nJAMS 4 configures the transport client to use its internal sniffing feature. This feature allows the transport client to automatically detect all data nodes in the cluster. It works as follows: The manually configured node address(es) is used for inital connection into the cluster only. When connected, the client resolves all data-node addresses (only) from the cluster and replaces the manually configured node address(es) with this resolved address list. Note that this resolved list does not necessarily contain the nodes configured manually! The internal nodes list is automatically refreshed in regular intervals (default is 5 seconds). The client then directs all requests directly to the data nodes that are responsible for serving a certain request. The client does not direct any requests to the master or client nodes. As a result, when using sniffing, there is no use for client nodes. Disabling Sniffing: Prior to nJAMS 4.1 sniffing can only be disabled by a system property: njams.indexer.sniffDisabled=true Since nJAMS 4.1 sniffing can be disabled both using the installer and the indexing configuration page in nJAMS UI. When sniffing is disabled, the transport client communicates only with the manually configured nodes in a round robin fashion, though most action will probably be two-hop operations. By default, sniffing is enabled. Please note: If you want to use sniffing in combination with SSL encryption, all nodes have to be configured to publish with an address that matches the address in the SSL certificate. Use network.publish_host: <full-DNS-name> in elasticsearch.yml of each node.
   Add Node	Add nodes of your Elasticsearch cluster, depending on configuration of sniffing. Enter nanme or IP address of an Elasticsearch node followed by port, e.g. elastic01:9200.
   Extended Client Settings	Open the extended client settings dialog.

3. Use X-Pack Features:

   

   Setting	Description
   Use client	Enable/disable client authentication If enabled, enter username and password.
   authentication	This features can only be used in connection with Use SSL/TLS.
   Use SSL/TLS	disabled: does not use encryption
   encryption	default system keystore: uses the default Java keystore custom keystore: enter your own keystore certificate file: full path to certificate file

4. Indexing settings can be adjusted after you have to Stop Client:

   

   Setting	Description
   Number of Shards	Number of Shards for new Indexes
   Number of Replicas	Total number of Replica Shards with regards to all indexes. Please note: changing number of Replica Shards may cause a lot of traffic. Depending on your data volume it may take some time. Excursion: The main purpose of Replicas is for failover. If the node holding a primary shard is not working, a replica is promoted to the role of primary.
   Refresh interval	Enter refresh interval in seconds
   Index name prefix	Enter prefix for index naming

5. The bulk processing can be configured with regards to the following settings:

   

   Setting	Description
   Indexing Threads	Number of threads indexing documents
   Bulk flush	Maximum number of actions, before triggering a flush. Default: 1000.
   actions
   Bulk flush time	Maximum number of seconds to wait, before triggering a flush. Default: 5.
   (secs)
   Bulk flush size	Maximum size of JSON request body, before triggering a flush. Default: 20 MB.
   (MB)
   Compress	Discard transitions of successful process executions. Default: enabled.
   successful	Leave this option to enabled, if you want to save storage.
   transactions	Disable this option, if you want to analyze the process execution path.
   Max activities	Maximum number of activities of a single process instance.
   Max payload	Maximum size of payloads in [MB].

   Note

   Do only change these values in (D) and (E) on advise of Integration Matters Support or if you are substantially familiar with Elasticsearch.

6. Search settings:

   

   Setting	Description
   Maximum search	Reduce number of search results in order to speed up queries to use the resources of the
   results	Elasticsearch cluster more efficiently
   Disable query	Disable query optimization, if you want to retrieve the exact number of results
   optimization

Add an Elasticsearch node:

In case you installed a blank nJAMS Server without any other options, the Indexer Client is not configured. An nJAMS Administrator has to create a connection to the Elasticsearch cluster. In order to create a connection for the Indexer Client it is required to have an Elasticsearch cluster in place. If nJAMS Server discovers an Elasticsearch cluster, the Indexer Client will create the nJAMS structure (mapping, etc.) within the cluster automatically.

Note

In case you want to use an already installed Elasticsearch Cluster (not installed by the nJAMS installer), scripting has to be enabled. Login to your Elasticsearch Node(s) machine(s) and edit <ES_HOME>/config/elasticsearch.yml. The following entries have to be included, respectively changed: script.indexed: on, script.inline: on.

An Elasticsearch cluster is assigned by entering the Elasticsearch Cluster Name and at least one Node Address. These values are mandatory to create a valid connection to an Elasticsearch cluster.

Note

An Elasticsearch cluster may consist of only 1 Node, but it is still called a cluster.

1. Make sure the Indexer Client is stopped.

2. Enter the cluster name and the hostname or IP Address of at least one Node of the Elasticsearch cluster. Client name and Management URL are optional. The Management URL contains a link to an external management tool for Elasticsearch, which is installed separately. There are several 3rd party tools available. In this example ‘Kopf’ is used.

3. If applicable, add further Node addresses

4. Make sure a connection can be established to the Elasticsearch cluster by testing the connection

5. Save the Indexer Client configuration, once it is tested properly

6. If the Management URL was provided, click on ‘OPEN Management Tool’ to get a deeper look inside of the Elasticsearch cluster. If you have installed ‘Cerebro’, for example, you are directed to the page of Cerebro, which provides you a detailed view into your Elasticsearch cluster

7. Revert current settings to previous settings.

You are now ready to start the Indexer Client!

Start Indexer Client:

Start the Indexer Client for nJAMS Server to work with Elasticsearch.

Index Management:

Index Management allows you to manage the indexes of your Elasticsearch Cluster. The main use case for Index Management is re-indexing existing nJAMS indexes into a new format. The following chapter describes why it is potentially required to re-index indexes and how to re-index an index.

How to re-index existing indexes:

Re-indexing is required, if you want to be able to search for event data of the past like payload, stacktrace, activity duration, etc. If you just want to search for process executions, filtered by time or domain object, re-indexing is not required.

In contrast, if you come from Elasticsearch 2, you have to re-index the indexes from v2 to v5 before you can migrate to Elasticsearch 6. In this scenario re-indexing is required! The following instructions describe re-indexation from v5 to v6 as a sample for re-indexation from v2 to v4.

Initial situation after upgrading your Elasticsearch Cluster from 6 to 7 and upgraded nJAMS Server to 5.1:

When the upgrade is done and nJAMS Server is started again, you can enter the “Index Management” page to see the current state of your indexes of your Elasticsearch Cluster. Go to Administration > Connections > Indexer > Tab Index Management:

1. REFRESH updates the current page to see the last changes to the indexes.

   HIDE non-nJAMS indexes only show indexes used by nJAMS.

   MAINTENANCE Mode enables you to re-index the indexes. As soon as you activate the Maintenance Mode, Indexer and Data Providers will stop. That means, when you are in Maintenance Mode, message processing is off, as well as searching for monitoring data.

   DELETE all nJAMS indexes removes all indexes related to nJAMS. Be careful, when you confirm the following confirmation prompt, all data will be lost. Please note, once all indexes have been removed, nJAMS Server will immediately start creating new indexes again.

   RELEASE WRITE-BLOCKS makes the index and index metadata writable.

2. Index state let you know, whether there are indexes left of previous version in your Elasticsearch

3. This is the list of indexes of your Elasticsearch Cluster used by nJAMS:

   Name is the name of the index

   Type indicates the index type, respectively the usage of the index

   Expiration indicates the date, when the index will expire

   Primary documents indicates the number of documents, which are relevant to re-indexing

   Size indicates the size of the index

   Estimated re-index duration is a rough estimation about how long a re-indexing of an index may take. Initially the estimated time is “unknown”. As soon as you re-indexed the first index, nJAMS Server can estimate roughly the time of the other indexes. Please note, re-indexing a data index is much more time consuming than re-indexing a statistics index, not only because of the amount of data, but because of the structure of the index. Since the duration will be estimated according to the index(es) you indexed, the estimated duration may lead to misleading results, depending on index type. For example, you indexed a data index first. Afterwards the remaining indexes are estimated according to that baseline. This approach results in overestimating the duration for re-indexing of statistic indexes, i.e. re-indexing the statistic indexes will be quicker than estimated. On the other hand, when you re-index a statistic index first, the duration of data indexes will be underestimated, i.e. re-indexing the data indexes will take more time than estimated. To be on the safe side, we recommend to re-index a data index first.

   Index version is the version of the index format

   Read only indicates the index is read only, no data can be inserted or updated in that index. When re-indexing a v5 index, a new index in format v6 is created. The source index in v5 will remain in read-only state.

   Actions let you re-index, remove, or detach the selected index. These actions are only available, when Maintenance Mode is active. Re-indexing is only available for supported index types. You can only delete nJAMS related indexes and “.scripts” indexes.

Procedure for re-indexing:

1. Enter Administration > Connections > Indexer > Tab Index Management

2. Activate Maintenance Mode

3. Select the index you want to get re-indexed

4. Click on the pen icon in column Actions, the following dialog opens:

   

   Source index name: name of the source index

   Target index name: new name for the re-indexed index. In most cases there is no need to change the proposed name.

   Delete source index: when this option is enabled, the source index will be removed, if re-indexing was successful. When you disable this option, the source index will remain in state read-only. Please note: if there are both indexes (source and target index) available, you will get duplicate results in nJAMS GUI. It is recommended to remove the source indexes immediately after re-indexing, so just leave this option enabled.

   Estimated duration: estimated period of time for re-indexing

   Remaining duration: time left

   Documents processed: number of document that have been processed during re-indexing

   Task-Id: this is the Id of the task in Elasticsearch.

   Result: expected to be successful, of course.

5. Start re-indexing

6. When re-indexing was successful, you can see the result here:

   

   There is now a new index in format v6 and the previous index has been removed.

7. Continue re-indexing with the next index.

Database

The database connection is configured during the installation of nJAMS Server. See the nJAMS Server Installation Manual for more information. The JDBC settings cannot be changed within nJAMS GUI.

Usually there is no need to change JDBC settings later. Nevertheless, you can modify the settings using WildFly’s Administration Console.

1. Enter the URL of WildFly Administration Console: http://<machine_name>:<admin_port>

2. Navigate to Configuration:Subsystems > Subsystem:Datasources > Type:Non-XA > Datasource:njamsPool

Make sure the JNDI datasource reference is always ‘njams’.

Note

Changing the database setting may break your nJAMS Instance. In case the new schema is empty, nJAMS Server will create all necessary objects on startup. Make also sure nJAMS Server is stopped before switching database. Select “Shutdown” from nJAMS Server Application Deployment page of the Administration Console.

JMS

JMS connections are referenced by the Data Providers. It is required to create at least one JMS connection. You may modify or delete existing JMS connections, or create new one. When selecting the JMS entry within the Connections category, a list of available connections is shown:

1. Select ‘JMS’ from Category Connections

2. List of configured JMS connections

3. Select an entry and you can see the details on the right hand side

ADD, EDIT a JMS connection:

4. Select one of the available JMS providers from the list. By choosing “Others” you can enter individual parameters for any other JMS provider.

5. Specify the following information

   Attributes	Description
   Name	Name of the JMS Connection
   JNDI	Activate this option, if you want to use JNDI
   Connection factory	For JNDI specify the connection factory according to your JMS Server
   Provider URL	For no JNDI enter URL of your JMS Server: tcp://<servername>:<port>
   Username	JMS user account for nJAMS Server
   Password	Password of the user
   Destination	This is the prefix of destinations used by nJAMS Server.

   Note

   If you want to use JNDI, you have to setup a JNDI context first. Afterwards you can specify this JNDI context within your JMS connection.

6. Enable fix response queues for communication with nJAMS Clients

7. If the JMS connection should use SSL, the following properties require configuration

   

   Attributes	Description
   Use SSL	check, to open the SSL configuration. Please note: SSL encryption is only available for TIBCO EMS
   SSL trace	check, to enable tracing when establishing an SSL connection
   SSL debug trace	check, to enable more verbose tracing
   Trusted Certificates	enter the path to a directory with trusted certificates (Note: remember that the path must be valid on the server’s host, not the local machine from where you access the web interface!)
   Expected Hostname	If set, the returned server name will be matched against this value. If the expected hostname and returned hostname do not match, the connection will not be established. Default: check <disabled>
   Public Key	Absolute path to the public key on the server’s host.
   Private Key password	The password for the private key
   Private Key	Absolute path to the private key on the server’s host.
   SSL Vendor	The SSL vendor to use (default j2se)
   SSL ciphers	The ciphers to use. Default: allow all.

8. Click on TEST connection to verify your connection.

9. If the connection could be established successfully, click on “SAVE”

To validate the settings made, press the “Test connection” button, at the bottom. Once the settings are validated, click on the “Save settings” button. “Cancel” discards all changes and closes the dialog.

DELETE a JMS connection:

Select one or multiple JMS connections and click on DELETE to remove the selected connections.

Note

nJAMS Server will prevent you from deleting JMS connections with reference to an existing Data Provider. Stop and Remove the Data Provider first, before deleting the JMS connection.

JNDI

Configure a JNDI context, if you want to use JMS with JNDI:

1. Select a JMS Provider

2. Specify a Name for the JNDI context and Username, Password. Furthermore, specify a valid Provider URL; you can also enter a fault-tolerant (failover) URL.

3. Save the JNDI context configuration.

LDAP

The LDAP feature allows you to configure a connection to a directory service. Through this connection, you can import either roles and associated users or users for existing roles in nJAMS Server or both. Authentication of an imported user will be done by the directory service. Authorization will be done via the access rights associated with the roles of the user.

The import of users and roles can be made ad hoc or scheduled via a configurable job. In both cases, all configured roles and users will be read from the directory service and created or updated in nJAMS Server. Roles or users which were formerly imported but do not longer match the configuration will be removed from nJAMS Server. See chapter ‘System Control > Jobs > LDAPSynchronization’ for learning how to configure a scheduled job for periodic import.

Roles imported from a directory service cannot be changed in nJAMS Server except for the granting and revoking of access right. Users must be linked to these roles in the directory service. They cannot be assigned to an imported role in nJAMS Server. Users imported from a directory service cannot be changed in nJAMS except for the roles to which the users is associated. An imported user can be added to or removed from a role created in nJAMS Server but not to a role imported from the directory service. This has to be done inside of the directory service. As the login will be done against the directory service via LDAP, the password cannot be changed in nJAMS Server.

To access a directory service you have to configure a LDAP connection by providing a host URL and optional a user name respectively a bind DN and a password. This depends on the setting of your directory service. After the connection has been configured you may test if the connection can be established.

If you want to import roles from LDAP, you need to create a LDAP query and attribute mappings for both the roles and their associated users. If you want to import users to already existing roles you need to create a LDAP query and a user attribute mapping on the LDAP tab of the role (see Receiving Users from a directory service).

For a LDAP query you need both a query string and a search base.

For importing roles, at least a mapping for the role name is required. A role comment may optionally be mapped.

You shall also give a user query string and at least a mapping for the user names to import users associated with the imported roles. The mapping of the first name, last name, email and a comment are not mandatory. Because these fields cannot be changed in nJAMS Server you will have to map these fields too, if needed.

You can test your query and mapping with the test button. You will see the roles from your directory service that matches your role query. Click on role name to see the users associated to that role for your user query and mapping:

When the test is successful, save the settings.

Due to the name of roles and users in nJAMS Server must be unique, the identification of a user or role always takes place by the mapped name and not by the distinguished name (DN). The DN of an imported user or role will be shown on the role or user page.

1. Activate starts synchronization of LDAP. Deactivate removes LDAP sychronization and asks to convert or remove synchronized users/roles.

   

   alt:

   Deactivate synchronization

LDAP over SSL:

You can use LDAP over SSL to secure your LDAP connection. Please prepare the following:

1. Make sure your LDAP Server is prepared to allow LDAP over SSL (LDAPS).

2. Export the Root CA Certificate as base64 encoded X.509 file, which signed the LDAP Server certificate

   In the LDAP dialog enter a LDAPS connection and either the path to the Truststore or the folder that contains the certificate:

   

   2. LDAP URL: Enter the LDAPS connection to your LDAP Server. As soon as you enter “ldaps”, the SSL Settings below are available for input.

   3. Truststore: enter full path to your trustrore file that contains the certificate.

   4. Certificates Folder: Alternatively enter the path to the folder that contains the certificate, e.g. /opt/njams/ldapcerts/.

Please restart nJAMS Server for the changes to take effect.

Email

The system may send email messages. For example, a user can request a password reset at the login page. This will trigger an email being sent to his mailbox, containing a URL to reset the password. To enable email notifications, the administrator must configure a SMTP connection. SMTP is the only protocol supported for email communication.

1. The SMTP connection must be configured as follows:

   Host: the name or IP address of the SMTP server

   Use STARTTLS: enable this option to encrypt traffic between the SMTP and nJAMS Server

   Port: port of the SMTP server

   Username: if the SMTP server requires authentication, enter a valid user name. Leave this field empty, if your SMTP server does not require authentication

   Password: the SMTP server user’s password

   Sender: the emails sent by nJAMS Server will use this value as the sender’s email address

   Note

   Emails cannot be sent to nJAMS Server

2. You can save the current settings once the test email could be sent successfully.

3. To send a test email enter the recipient email address and click on SEND. A test email should be received in the recipient’s inbox.

Argos

The Argos configuration page allows you to view the state of the Argos component and configure Argos.

1. State of the Argos component

   Toggle Argos: enables / disables Argos within nJAMS Server

   Argos component state: current state of Argos

   Rules Manager: state of the Rules Manager component of Argos

   Data Provider: state of the Argos Data Provider

   Indexer: state of the Indexer

   Active Indexing Threads: Total number of the indexing threads

2. Configuration Argos database

   Argos comes with an embedded graph database to store metrics data. There is no need to make any changes to the configuration of the Argos database.

3. General settings

   History retention: enter the retention period of the history information. Default is 14 days.

4. Subagent

   nJAMS Server has a built-in Subagent to report metrics of the JVM of WildFly, respectively of nJAMS Server. Metrics from Subagent are sent to nJAMS Agent.

   Enable JVM statistics: enable / disable Subagent. Subagent is enabled by default.

   nJAMS Agent IP address: enter the IP address of the machine that runs nJAMS Agent.

   nJAMS Agent port number: enter the port number of nJAMS Agent. Default is 6450.