Skip to main content
ComponentsEnterprise
Availability2014.1 or later

Elasticsearch Usage

Storage and search roles

Prepare Elasticsearch

Install and configure cluster

Migrate Existing Data

Move historical data over

Production Validation

Checks before go-live

​
Overview

Elasticsearch is the preferred datastore for on-premise GermainUX deployments. Its distributed search and analytics architecture is well suited to:
  • High-volume, near-real-time telemetry ingestion
  • Time-series monitoring data
  • Fast filtering and aggregation
  • Full-text search
  • Large dashboards and investigative queries
  • Session Replay and technical traces
  • Horizontal scaling
  • Time-based retention
A managed Elasticsearch service can also be used when your organization does not want to operate the cluster directly.

​
Before You Begin

Confirm with Germain Software:
  • The Elasticsearch version supported by your GermainUX release
  • Required Elasticsearch plugins and settings
  • Expected daily ingestion volume
  • Raw and aggregated data-retention periods
  • Session Replay volume
  • Node and storage requirements
  • High-availability requirements
  • Authentication and TLS configuration
  • Whether existing GermainUX data must be migrated
Do not install or upgrade Elasticsearch independently of the GermainUX compatibility requirements.

​
How GermainUX Uses Elasticsearch

​
Raw Data

Raw indexes retain the complete monitoring facts collected by GermainUX. Raw data is required for calculations that cannot be reliably reconstructed from aggregated data, including:
  • Percentiles
  • Standard deviation
  • Detailed root-cause analysis
  • Instance-level investigation
  • Technical tracing
  • Other calculations that require the original distribution
After raw data expires, these calculations may no longer be available for the affected period.

​
Aggregated Data

GermainUX aggregation is similar to Elasticsearch rollups: detailed facts are summarized into larger time intervals to reduce long-term storage. GermainUX can aggregate data into hourly indexes that may be retained longer than the corresponding raw data. Aggregated indexes support measures such as:
  • Minimum
  • Maximum
  • Average
  • Sum
  • Count
Aggregated data generally requires less storage but cannot replace raw data for every calculation.

​
Unified Queries

GermainUX can query raw and aggregated indexes through a single request and merge the results into one view. This allows recent periods to use detailed raw data while older periods use retained aggregates, subject to the measures supported by each data type.

​
Time-Series Data

GermainUX time-series data streams are primarily append-only. For fact data streams, GermainUX can use a lifecycle such as:
  1. New data is written to a hot index.
  2. After the configured hot period—typically one day—the index becomes read-only or moves to colder storage.
  3. Raw data remains available until its retention period expires.
  4. Aggregated indexes may be retained for a longer period.
The exact lifecycle must be adjusted to your storage, performance, and retention requirements.

​
Architecture Requirements

A GermainUX deployment using Elasticsearch generally requires: Production environments should also include:
  • TLS
  • Authentication and role-based access
  • Backups or snapshots
  • Monitoring
  • Retention policies
  • Sufficient operational free space
  • Replication and high availability when required

​
Deployment Scenarios

​
New GermainUX Deployment

For a new deployment:
1

Prepare the Elasticsearch cluster

2

Configure security, networking, storage, and snapshots

3

Install the GermainUX search indexes and templates

4

Configure GermainUX Enterprise to connect to Elasticsearch

5

Initialize the required GermainUX data structures

6

Validate connectivity and query execution

7

Start monitoring with a limited data volume

8

Confirm ingestion, search, aggregation, retention, and performance

9

Expand to production volume

​
Migration From Another Datastore or Environment

When migrating existing data:
1

Prepare the target Elasticsearch cluster

2

Export the GermainUX configuration from the source environment

3

Create a temporary GermainUX Enterprise instance

4

Connect the temporary instance to a separate SQL configuration database

5

Configure unique queue names so it does not interfere with the existing environment

6

Configure an appropriate Elasticsearch index prefix

7

Initialize the Elasticsearch data structures

8

Configure the migration tool

9

Migrate and validate each data type

10

Switch the target GermainUX environment only after validation is complete

Migration should be planned with Germain Software because the required scripts and sequence depend on the source datastore, GermainUX version, data volume, and retention requirements.

​
Prepare Elasticsearch

​
1 1. Provision the Infrastructure

Provision the required Elasticsearch nodes and storage based on:
  • Daily AA Unit volume
  • Peak ingestion rate
  • Total retained raw data
  • Aggregated-data retention
  • Session Replay
  • Replication
  • Query workload
  • Expected growth
Use SSD storage with sufficient IOPS, throughput, and low latency. See:

​
2. Configure Time

Configure Elasticsearch and GermainUX hosts with consistent:
  • Time zone
  • System time
  • Network Time Protocol synchronization
Time inconsistencies can affect index boundaries, retention, correlation, and time-based analysis.

​
3. Configure Networking

Allow only the required communication between:
  • GermainUX Enterprise and Elasticsearch
  • Authorized administrators and Elasticsearch
  • Kibana and Elasticsearch, when Kibana is used
  • Elasticsearch nodes within the cluster
  • Snapshot repositories, when configured
Do not expose Elasticsearch or Kibana directly to untrusted networks.

​
4. Configure Security

Before production use:
  • Enable TLS for client and node communication.
  • Enable authentication.
  • Create dedicated GermainUX service accounts.
  • Apply least-privilege roles.
  • Restrict network access.
  • Protect credentials outside source-controlled configuration.
  • Configure audit logging when required.
  • Secure Kibana if it is installed.
See Securing Elasticsearch and Kibana.

​
5. Configure Backups

Configure Elasticsearch snapshots to an approved repository. Validate:
  • Snapshot scheduling
  • Retention
  • Repository permissions
  • Encryption
  • Off-site or cross-region copies where required
  • Restoration procedures
A snapshot is not considered a usable backup until a restore has been tested.

​
Configure GermainUX Enterprise

​
Temporary Migration Instance

A temporary GermainUX Enterprise instance may be required to prepare Elasticsearch before migrating existing data. It requires:
  • A SQL database for GermainUX configuration
  • Access to the target Elasticsearch cluster
  • Access to the required GermainUX packages and scripts
  • A unique ActiveMQ queue prefix
  • A unique Elasticsearch index prefix
No production monitoring services should run on the temporary instance.

​
Update common.properties

Configure the applicable properties in:
For a migration environment:
  1. Assign unique queue names or prefixes so the temporary instance can share ActiveMQ without consuming production messages.
  2. Set a meaningful Elasticsearch index prefix, such as one identifying the environment.
  3. Configure the Elasticsearch URL and connection settings.
  4. Configure authentication and TLS.
  5. Configure the SQL configuration database.
  6. Validate credentials and network connectivity.
Example property:
Use a distinct prefix for each development, QA, staging, or production environment sharing the same Elasticsearch cluster.

​
Initialize the Datastore

​
1. Install Search Index Definitions

Import the GermainUX Elasticsearch index definitions from:
Use the files supplied with the exact GermainUX release being deployed. Do not reuse index definitions from a different release unless Germain Software confirms compatibility.

​
2. Start GermainUX Enterprise

Start the temporary or target GermainUX Enterprise instance and verify that it connects successfully to:
  • The SQL configuration database
  • Elasticsearch
  • ActiveMQ
Review the logs for authentication, certificate, template, index, or connection errors.

​
3. Apply the Remaining Elasticsearch Configuration

Use the GermainUX REST endpoint or deployment script provided for your release to create the remaining:
  • Index templates
  • Data streams
  • Lifecycle configuration
  • Mappings
  • Aliases
  • Required GermainUX metadata
Contact Germain Software for the correct script, endpoint, and parameters.

​
Migrate Existing Data

​
Configure the Migration Tool

Update the migration tool’s YAML configuration with:
  • Source datastore connection
  • Target Elasticsearch connection
  • Authentication
  • TLS settings
  • Index prefix
  • Source and target environment
  • Date range
  • Batch size
  • Data type
  • Logging and output locations
Protect database and Elasticsearch credentials according to your security requirements.

​
Run the Migration

Run the migration once for each supported data type using the examples provided with the migration tool. Independent data types may be migrated in parallel when:
  • The source datastore can support the additional load.
  • Elasticsearch can support the combined ingestion rate.
  • The migration tool and version allow parallel execution.
  • Sufficient CPU, memory, network, and storage capacity are available.
Monitor:
  • Source-database load
  • Migration throughput
  • Elasticsearch indexing latency
  • Rejected writes
  • Disk utilization
  • Cluster health
  • Error and retry counts

​
Validate Migrated Data

For every data type, compare:
  • Record counts
  • Minimum and maximum timestamps
  • Application and environment values
  • Daily volumes
  • Key measures and pivots
  • Representative raw facts
  • Aggregated results
  • Dashboard results
  • Session Replay availability, when applicable
Do not remove or modify the source data until the migration has been accepted.

​
Validate Elasticsearch Queries

GermainUX can monitor and validate Elasticsearch queries. To configure query monitoring:
1

Open Germain Workspace

2

Go to System > Engine Settings > Component Types

3

Click Add

4

Select Elasticsearch Query Monitor Component

5

Complete the configuration

6

Validate that the component can connect and execute its configured query

Use query validation to confirm:
  • Connectivity
  • Authentication
  • TLS
  • Index access
  • Query syntax
  • Expected results
  • Query performance

​
Production Validation

Before production rollout, confirm that:
  • The Elasticsearch version is supported.
  • All cluster nodes are healthy.
  • TLS and authentication are enabled.
  • GermainUX uses a dedicated service account.
  • Index templates and mappings are installed.
  • The correct environment prefix is configured.
  • Raw data is being indexed.
  • Aggregated data is being generated.
  • Raw and aggregated queries return expected results.
  • Percentiles and standard deviation work while raw data is retained.
  • Lifecycle and retention policies operate correctly.
  • Snapshot and restore procedures have been tested.
  • Disk, heap, ingestion, and query performance are monitored.
  • Sufficient capacity remains for merges, reindexing, and growth.
  • Migrated data has been reconciled with the source.
For the required Elasticsearch scripts, migration tools, or deployment assistance, contact Germain Software.
Create new component on Component Types page
Elasticsearch Query Monitor Component
Contact Germain Software to discuss an on-premise deployment or request assistance.

​
Get Help

Contact GermainUX Support

The GermainUX team helps you get up and running faster, empower your team, turn data into actionable insights, and optimize performance to deliver measurable business outcomes.

Can't find what you need?

Search all GermainUX docs with Ctrl + K, or use the feedback buttons at the bottom of this page to tell us what is missing.

The content of this document is proprietary to GERMAIN SOFTWARE.