Skip to main content
ComponentsEngine
Availability2022.1 or later

What Is Monitored

Cluster, nodes, shards

How It Works

HTTP and query monitors

Deploy

One wizard per feature

KPIs

Cluster health with SLAs

Overview

GermainUX watches the Elasticsearch clusters your applications search and index into. You see the cluster status, the number of nodes and shards, and the CPU, memory, disk and JVM heap of every node. KPIs with default SLAs and an Elasticsearch Overview dashboard come ready to use. A cluster that turns yellow or red, a node that leaves, or a disk that nears its watermark raises an alert before searches start to fail. This page covers Elasticsearch clusters that you run for your own applications. To set up Elasticsearch as the GermainUX datastore, see Elasticsearch On Premise.

Cluster Status over Time

Green, yellow or red for every poll, with an SLA that warns on yellow and turns red on red.

Every Node Side by Side

CPU, memory, disk, JVM heap and thread count for each node in the cluster.

Disk Before It Blocks Writes

An event when a node’s disk passes the cluster’s low disk watermark.

Your Own Queries as KPIs

Run Elasticsearch SQL or Query DSL on a schedule, and trend what the queries return.

Questions It Answers

  • Is the cluster green right now, and when was it last yellow or red?
  • Which node left the cluster, and for how long?
  • Which node is running short of JVM heap or disk?
  • Are all shards active again after a restart?
  • Is each rollup job still processing documents?
  • Does the cluster’s REST endpoint answer from the Engine?

What Is Monitored

Analytics, Alerts and Automation

AI Analysis

Ask AI Explore questions such as “Which Elasticsearch node had the highest heap usage this week?” Outlier detection flags a node whose heap, CPU or disk usage moves outside its normal range.

Dashboards and Reports

In the left menu, go to Dashboards > All, and then open the Elasticsearch Overview dashboard. It shows the cluster status over time, the CPU, memory, JVM heap and disk usage of each node, read and write operations, thread count, and the documents processed by rollup jobs. You can build your own dashboards and reports on the same KPIs. The node charts show the monitors whose application is Elasticsearch. Use that application name when you deploy the monitor.
Elasticsearch Overview dashboard with the cluster status turning yellow, the JVM heap usage of node es-03 rising, and CPU, memory, disk, read and write operations for nodes es-01 to es-03

Elasticsearch Overview Dashboard - GermainUX

Click a chart to open its KPI in the Analysis screen. The analysis compares the selected period with the previous one and lists the clusters and nodes that changed most.
Analysis screen for the Elasticsearch Cluster Health KPI during the es-02 restart, with an average of 84.722% against 100.000% on the previous days, and a dip in the relationships chart while the cluster is red

Analysis of Elasticsearch Cluster Health - GermainUX

Analysis

Each walkthrough below opens one KPI in the Analysis screen. The top row compares the selected hours with the same hours on the previous days, and shows the trend. The relationships chart shows the KPI over the days around the selected period, which is highlighted. The segmentation breakdown lists the leading factors, and Show all segments lists each node, cluster and endpoint with its own value.

Investigate Heap Pressure on a Node

When searches slow down or a node stops answering, open the Java Heap Usage KPI. The average across all nodes can hide one node in trouble, so look at the leading factors, and click Show all segments to compare the nodes under By System Name. In the example below, a dashboard runs wide aggregations on search-prod, and the heap of es-03 climbs past the 75% SLA. Between 08:00 and 10:00 UTC on 11 November, the average heap usage is 51.016%, up 12.1% on the previous days. The breakdown marks es-03 as degraded, at 65.188% and up 45.1%, while es-01 and es-02 stay near 46%. es-03 has 16 readings instead of 24, because it then drops out of the cluster. The SLA on Java Heap Usage raises an alert while the heap is still climbing.
Analysis screen for the Java Heap Usage KPI filtered to the Elasticsearch application, with an average of 51.016% up 12.1% on the previous days, and es-03 marked as degraded at 65.188% next to es-01 and es-02 at about 46%

Analysis of Java Heap Usage - GermainUX

Investigate a Node That Leaves the Cluster

When the cluster turns yellow, open the Elasticsearch Cluster Node Count KPI. The top row shows how many nodes the cluster had on average, against the previous days, and the hourly trend shows when a node left and came back. In the example below, es-03 drops out of the cluster after the heap pressure above. The cluster is yellow while Elasticsearch rebuilds the replicas of es-03 on es-01 and es-02. Between 07:00 and 12:00 UTC on 11 November, the cluster averages 2.850 nodes against 3.000 on the previous days, down 5.0%. The hourly trend dips and recovers when es-03 rejoins, about 45 minutes later.
Analysis screen for the Elasticsearch Cluster Node Count KPI, with an average of 2.850 nodes against 3.000 on the previous days, and a dip in the hourly trend and the relationships chart

Analysis of Elasticsearch Cluster Node Count - GermainUX

Investigate a Disk Watermark

When a node’s disk passes the low disk watermark, Elasticsearch Errors records an event, and Elasticsearch stops placing new shards on that node. Open the Disk Usage KPI to see which node is filling up and how fast. Click Show all segments to compare the nodes under By System Name. In the example below, log indices that are never deleted fill the disk of es-02 past the 85% low watermark. Between 05:00 and 16:00 UTC on 13 November, the average disk usage is 64.408%, up 6.8% on the previous days. Under By System Name, es-02 averages 73.550%, up 18.0%, while es-01 and es-03 stay near 59% and 61%. The line of es-02 climbs for several hours, and drops once the old indices are deleted. The Disk Usage SLA of 75% warns you before the node reaches the watermark.
Analysis screen for the Disk Usage KPI filtered to the Elasticsearch application, with an average of 64.408% up 6.8% on the previous days, and es-02 at 73.550% up 18.0% under By System Name, next to es-01 and es-03

Analysis of Disk Usage - GermainUX

Investigate a Red Cluster After a Restart

When the cluster turns red, some primary shards have no copy on any node. Open the Elasticsearch Cluster Health KPI, as in the analysis under Dashboards and Reports, to see how long the cluster stayed red. Then open Elasticsearch Active Shards to see how many shards were missing, and whether all of them are active again. In the example below, es-02 restarts for patching at night. The audit index has two primary shards and no replicas, both on es-02, so the cluster stays red until es-02 is back. Between 22:00 UTC on 13 November and 04:00 UTC on 14 November, the cluster averages 86.125 active shards, against 88.000 on the previous days, down 2.1%. The hourly trend dips while es-02 is away, and returns to all shards active once it rejoins.
Analysis screen for the Elasticsearch Active Shards KPI, with an average of 86.125 active shards against 88.000 on the previous days, and a dip in the hourly trend and the relationships chart

Analysis of Elasticsearch Active Shards - GermainUX

Investigate an Unreachable Cluster Endpoint

When the Elasticsearch charts stop updating, open the HTTP Availability KPI. It shows whether the cluster’s REST endpoint answered each poll. While the endpoint doesn’t answer, the other Elasticsearch KPIs get no values. In the example below, the load balancer search-prod-lb doesn’t answer for 25 minutes during network maintenance. Between 06:00 and 11:00 UTC on 15 November, availability averages 91.667%, against 100.000% on the previous days, down 8.3%. The breakdown marks search-prod-lb as degraded, at 0.000% for its 5 failed polls. The HTTP Availability SLA of 95% raises an alert when the endpoint stops answering.
Analysis screen for the HTTP Availability KPI filtered to the Elasticsearch application, with an average of 91.667% against 100.000% on the previous days, a dip in the relationships chart, and search-prod-lb marked as degraded at 0.000%

Analysis of HTTP Availability - GermainUX

SLAs and Alerts

The cluster KPIs come with default SLAs, listed under KPIs. Get an alert by email, SMS, Slack or Teams when the cluster turns yellow or red, when a node’s heap or disk stays over its limit, or when the endpoint stops answering.

Automation

When an SLA is breached, an automated action can run a diagnostic script, call an HTTP endpoint or open an incident.

Correlation with Other Technologies

Correlate the cluster’s health with the applications that use it, such as a Java service whose searches slow down while a node is short of heap.

How It Works

Elasticsearch REST APIElasticsearch HTTP MonitorElasticsearch rulesKPIsAlerts
One HTTP monitor covers the whole cluster, because the cluster APIs answer for every node. Point it at any node, or at the load balancer in front of the cluster.

Before You Start

  • A GermainUX Engine can reach the cluster’s REST port, usually 9200. See GermainUX Engine.
  • If the cluster has security turned on, an Elasticsearch user for GermainUX. For cluster monitoring, it needs the monitor cluster privilege and the monitor index privilege. For the Query Monitor, it also needs read on the indices you query.
  • That user is saved as credentials in Settings > Data Sources > Credentials. See Data Sources.

Deploy

1

Open the HTTP Scenario wizard

In the left menu, go to Settings > Wizards. On the HTTP Scenario Component Deployment card, click Add.
2

Choose the cluster endpoint

On the Server step, fill in the fields:
3

Select the Elasticsearch HTTP Monitor

On the Component step, set Component Type to Elasticsearch HTTP Monitor.If you selected credentials, click Show Advanced, and set Auth Policy to Basic. Without an auth policy, GermainUX doesn’t send the credentials to the cluster.
HTTP Scenario Component Deployment wizard on the Component step, with Elasticsearch HTTP Monitor selected as the Component Type

HTTP Scenario Component Deployment Wizard for Elasticsearch - GermainUX

4

Deploy the monitor

On the Deployment step, fill in the fields, and finish the wizard:

Validate

1

Check the cluster from the Engine

On the Engine server, call the cluster health API:
Replace ES_HOST with the host in the URI data source, and ES_USER with the Elasticsearch user. Leave out -u ES_USER if the cluster has security turned off. The response shows the cluster status that GermainUX reports.
2

Find the cluster in GermainUX

After the next scheduled run, open the Elasticsearch Overview dashboard. Cluster Health shows the cluster’s status, and the node charts show each node. For the Query Monitor, open the KPI that your rule writes to.

Troubleshooting

Check that the Engine reaches the URI with the curl command under Validate, that the Component Type is Elasticsearch HTTP Monitor, and that the scheduled time has passed. With the default schedule, the first poll can take up to an hour.
Select the Elasticsearch user in Credentials, and set Auth Policy to Basic under Show Advanced on the Component step. Check that the user has the privileges listed under Before You Start.
The node charts of Elasticsearch Overview show only monitors whose application is Elasticsearch. Set Application Name to Elasticsearch on the HTTP monitor.
The query component needs a rule that turns its results into KPI values. Assign one with Use Custom Rule? on the component, as in the Query Monitor steps under Deploy.
Without a URL, the Query Monitor connects over plain HTTP to the database’s host and port. Select Configure with URL? on the database data source, and enter the https:// URL in JDBC URL.

Security and Privacy

Elasticsearch monitoring data includes:
  • Infrastructure identifiers: the cluster name, and the name and IP address of each node
  • Query results: the Query Monitor stores the fields that your queries return and your rule maps. Indexed documents can hold personal data or free text, so treat these results as possibly containing personal or customer data.
To mask, anonymize or exclude this data, see Data Privacy.

KPIs

The HTTP monitor also feeds these shared KPIs, which GermainUX uses for other technologies too: Adjust the thresholds to your cluster. See KPIs and SLAs.

Synthetic HTTP Execution

How HTTP monitors run on a schedule

Create a Rule

Turn query results into KPIs

Automation

More Automation Features

Get Help

The GermainUX team can help you write Query Monitor rules for your indices, and tune the SLAs to your cluster.

Contact GermainUX Support

The GermainUX team helps you get up and running faster, empower your team, turn data into actionable insights, and optimize performance to deliver measurable business outcomes.

Can't Find What You Need?

Search all GermainUX docs with Ctrl + K, or use the feedback buttons at the bottom of this page to tell us what is missing.

The content of this document is proprietary to GERMAIN SOFTWARE.