Elasticsearch - OnPremise (Preferred)

🔍 Overview

Elasticsearch is the preferred datastore for on-premise GermainUX deployments.

Its distributed search and analytics architecture is well suited to:

A managed Elasticsearch service can also be used when your organization does not want to operate the cluster directly.

🤝 Before You Begin

Confirm with Germain Software:

Do not install or upgrade Elasticsearch independently of the GermainUX compatibility requirements.

⚙️ How GermainUX Uses Elasticsearch

📥 Raw Data

Raw indexes retain the complete monitoring facts collected by GermainUX.

Raw data is required for calculations that cannot be reliably reconstructed from aggregated data, including:

After raw data expires, these calculations may no longer be available for the affected period.

📊 Aggregated Data

GermainUX aggregation is similar to Elasticsearch rollups: detailed facts are summarized into larger time intervals to reduce long-term storage.

GermainUX can aggregate data into hourly indexes that may be retained longer than the corresponding raw data.

Aggregated indexes support measures such as:

Aggregated data generally requires less storage but cannot replace raw data for every calculation.

🔗 Unified Queries

GermainUX can query raw and aggregated indexes through a single request and merge the results into one view.

This allows recent periods to use detailed raw data while older periods use retained aggregates, subject to the measures supported by each data type.

⏱ Time-Series Data

GermainUX time-series data streams are primarily append-only.

For fact data streams, GermainUX can use a lifecycle such as:

1. New data is written to a hot index.

2. After the configured hot period—typically one day—the index becomes read-only or moves to colder storage.

3. Raw data remains available until its retention period expires.

4. Aggregated indexes may be retained for a longer period.

The exact lifecycle must be adjusted to your storage, performance, and retention requirements.

🏢 Architecture Requirements

A GermainUX deployment using Elasticsearch generally requires:

Production environments should also include:

🚀 Deployment Scenarios

🆕 New GermainUX Deployment

For a new deployment:

1. Prepare the Elasticsearch cluster.

2. Configure security, networking, storage, and snapshots.

3. Install the GermainUX search indexes and templates.

4. Configure GermainUX Enterprise to connect to Elasticsearch.

5. Initialize the required GermainUX data structures.

6. Validate connectivity and query execution.

7. Start monitoring with a limited data volume.

8. Confirm ingestion, search, aggregation, retention, and performance.

9. Expand to production volume.

🔁 Migration From Another Datastore or Environment

When migrating existing data:

1. Prepare the target Elasticsearch cluster.

2. Export the GermainUX configuration from the source environment.

3. Create a temporary GermainUX Enterprise instance.

4. Connect the temporary instance to a separate SQL configuration database.

5. Configure unique queue names so it does not interfere with the existing environment.

6. Configure an appropriate Elasticsearch index prefix.

7. Initialize the Elasticsearch data structures.

8. Configure the migration tool.

9. Migrate and validate each data type.

10. Switch the target GermainUX environment only after validation is complete.

Migration should be planned with Germain Software because the required scripts and sequence depend on the source datastore, GermainUX version, data volume, and retention requirements.

🛠 Prepare Elasticsearch

1️⃣ 1. Provision the Infrastructure

Provision the required Elasticsearch nodes and storage based on:

Use SSD storage with sufficient IOPS, throughput, and low latency.

See:

🕒 2. Configure Time

Configure Elasticsearch and GermainUX hosts with consistent:

Time inconsistencies can affect index boundaries, retention, correlation, and time-based analysis.

📡 3. Configure Networking

Allow only the required communication between:

Do not expose Elasticsearch or Kibana directly to untrusted networks.

🔒 4. Configure Security

Before production use:

See Securing Elasticsearch and Kibana.

💾 5. Configure Backups

Configure Elasticsearch snapshots to an approved repository.

Validate:

A snapshot is not considered a usable backup until a restore has been tested.

💻 Configure GermainUX Enterprise

🔧 Temporary Migration Instance

A temporary GermainUX Enterprise instance may be required to prepare Elasticsearch before migrating existing data.

It requires:

No production monitoring services should run on the temporary instance.

✏️ Update common.properties

Configure the applicable properties in:

common.properties

For a migration environment:

1. Assign unique queue names or prefixes so the temporary instance can share ActiveMQ without consuming production messages.

2. Set a meaningful Elasticsearch index prefix, such as one identifying the environment.

3. Configure the Elasticsearch URL and connection settings.

4. Configure authentication and TLS.

5. Configure the SQL configuration database.

6. Validate credentials and network connectivity.

Example property:

germain.elastic.properties.indexPrefix

Use a distinct prefix for each development, QA, staging, or production environment sharing the same Elasticsearch cluster.

🚚 Initialize the Datastore

📁 1. Install Search Index Definitions

Import the GermainUX Elasticsearch index definitions from:

servicesDistro/install/indexer/es

Use the files supplied with the exact GermainUX release being deployed.

Do not reuse index definitions from a different release unless Germain Software confirms compatibility.

▶️ 2. Start GermainUX Enterprise

Start the temporary or target GermainUX Enterprise instance and verify that it connects successfully to:

Review the logs for authentication, certificate, template, index, or connection errors.

⚙️ 3. Apply the Remaining Elasticsearch Configuration

Use the GermainUX REST endpoint or deployment script provided for your release to create the remaining:

Contact Germain Software for the correct script, endpoint, and parameters.

☁️ Migrate Existing Data

⚙️ Configure the Migration Tool

Update the migration tool’s YAML configuration with:

Protect database and Elasticsearch credentials according to your security requirements.

▶️ Run the Migration

Run the migration once for each supported data type using the examples provided with the migration tool.

Independent data types may be migrated in parallel when:

Monitor:

✅ Validate Migrated Data

For every data type, compare:

Do not remove or modify the source data until the migration has been accepted.

🔎 Validate Elasticsearch Queries

GermainUX can monitor and validate Elasticsearch queries.

To configure query monitoring:

1. Open Germain Workspace.

2. Go to System > Engine Settings > Component Types.

3. Click Add.

4. Select Elasticsearch Query Monitor Component.

5. Complete the configuration.

6. Validate that the component can connect and execute its configured query.

Use query validation to confirm:

📋 Production Validation

Before production rollout, confirm that:

For the required Elasticsearch scripts, migration tools, or deployment assistance, contact Germain Software.

Contact Germain Software to discuss an on-premise deployment or request assistance.