> ## Documentation Index
> Fetch the complete documentation index at: https://docs.germainux.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Elasticsearch Monitoring

> Know the moment your Elasticsearch cluster turns yellow or red, and which node is short of heap or disk.

Components
[`Engine`](https://docs.germainux.com/deploy/germain-ux-engine)

Availability
2022.1 or later

> [card] [What Is Monitored](#what-is-monitored)
>
> Cluster, nodes, shards

> [card] [How It Works](#how-it-works)
>
> HTTP and query monitors

> [card] [Deploy](#deploy)
>
> One wizard per feature

> [card] [KPIs](#kpis)
>
> Cluster health with SLAs

## Overview

GermainUX watches the Elasticsearch clusters your applications search and index into. You see the cluster status, the number of nodes and shards, and the CPU, memory, disk and JVM heap of every node. KPIs with default SLAs and an **Elasticsearch Overview** dashboard come ready to use. A cluster that turns yellow or red, a node that leaves, or a disk that nears its watermark raises an alert before searches start to fail.

This page covers Elasticsearch clusters that you run for your own applications. To set up Elasticsearch as the GermainUX datastore, see [Elasticsearch On Premise](https://docs.germainux.com/deploy/deployment-on-premise/elasticsearch-database-preferred.md).

> [card] Cluster Status over Time
>
> Green, yellow or red for every poll, with an SLA that warns on yellow and turns red on red.

> [card] Every Node Side by Side
>
> CPU, memory, disk, JVM heap and thread count for each node in the cluster.

> [card] Disk Before It Blocks Writes
>
> An event when a node's disk passes the cluster's low disk watermark.

> [card] Your Own Queries as KPIs
>
> Run Elasticsearch SQL or Query DSL on a schedule, and trend what the queries return.

### Questions It Answers

* Is the cluster green right now, and when was it last yellow or red?
* Which node left the cluster, and for how long?
* Which node is running short of JVM heap or disk?
* Are all shards active again after a restart?
* Is each rollup job still processing documents?
* Does the cluster's REST endpoint answer from the Engine?

## What Is Monitored

| Area | What GermainUX monitors |
| - | - |
| **Cluster health** | The cluster status (green, yellow or red), the number of active shards and the number of nodes |
| **Shards** | The total number of shards in the cluster, and the number of shards on each node |
| **Nodes** | CPU, memory, disk and swap usage, JVM heap usage, thread count, and read and write operations of each node |
| **Disk watermark** | An event when a node's disk usage passes the cluster's low disk watermark, `85%` unless the cluster sets another value |
| **Rollup jobs** | The documents each rollup job has processed |
| **Availability** | Whether the cluster's REST endpoint answers each poll |
| **Query results** | The values your own Elasticsearch SQL or Query DSL queries return, with the Elasticsearch Query Monitor |
| **Host** | CPU, memory, disk and network of the servers the nodes run on, and whether the Elasticsearch process is running. See [Linux monitoring](https://docs.germainux.com/integrations/linux-redhat-suse-monitoring.md) or [Windows monitoring](https://docs.germainux.com/integrations/windows-monitoring.md). |
| **Logs** | Errors and warnings from the Elasticsearch logs, with [File and Log Monitoring](https://docs.germainux.com/integrations/file-log-monitoring.md) |

## Analytics, Alerts and Automation

### AI Analysis

Ask [AI Explore](https://docs.germainux.com/analytics/ai-driven-insights.md) questions such as "Which Elasticsearch node had the highest heap usage this week?" [Outlier detection](https://docs.germainux.com/analytics/outlier-detection.md) flags a node whose heap, CPU or disk usage moves outside its normal range.

### Dashboards and Reports

In the left menu, go to **Dashboards > All**, and then open the **Elasticsearch Overview** dashboard. It shows the cluster status over time, the CPU, memory, JVM heap and disk usage of each node, read and write operations, thread count, and the documents processed by rollup jobs. You can build your own [dashboards and reports](https://docs.germainux.com/dashboards/dashboard-and-report.md) on the same KPIs.

The node charts show the monitors whose application is `Elasticsearch`. Use that application name when you [deploy](#deploy) the monitor.

![Elasticsearch Overview dashboard with the cluster status turning yellow, the JVM heap usage of node es-03 rising, and CPU, memory, disk, read and write operations for nodes es-01 to es-03](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-overview-dashboard.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=cef79aa1b65ab5c0a3e57851587f2c72)

Elasticsearch Overview Dashboard - GermainUX

Click a chart to open its KPI in the [Analysis](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md) screen. The analysis compares the selected period with the previous one and lists the clusters and nodes that changed most.

![Analysis screen for the Elasticsearch Cluster Health KPI during the es-02 restart, with an average of 84.722% against 100.000% on the previous days, and a dip in the relationships chart while the cluster is red](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-cluster-health-analysis.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=7820b618bd1d1796689602d651816e5e)

Analysis of Elasticsearch Cluster Health - GermainUX

### Analysis

Each walkthrough below opens one KPI in the [Analysis](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md) screen. The top row compares the selected hours with the same hours on the previous days, and shows the trend. The relationships chart shows the KPI over the days around the selected period, which is highlighted. The segmentation breakdown lists the leading factors, and **Show all segments** lists each node, cluster and endpoint with its own value.

### Investigate Heap Pressure on a Node

When searches slow down or a node stops answering, open the **Java Heap Usage** KPI. The average across all nodes can hide one node in trouble, so look at the leading factors, and click **Show all segments** to compare the nodes under **By System Name**.

In the example below, a dashboard runs wide aggregations on `search-prod`, and the heap of **es-03** climbs past the `75%` SLA. Between 08:00 and 10:00 UTC on 11 November, the average heap usage is 51.016%, up 12.1% on the previous days. The breakdown marks **es-03** as degraded, at 65.188% and up 45.1%, while **es-01** and **es-02** stay near 46%. **es-03** has 16 readings instead of 24, because it then drops out of the cluster. The [SLA](https://docs.germainux.com/analytics/slas.md) on **Java Heap Usage** raises an [alert](https://docs.germainux.com/automation/notifications.md) while the heap is still climbing.

![Analysis screen for the Java Heap Usage KPI filtered to the Elasticsearch application, with an average of 51.016% up 12.1% on the previous days, and es-03 marked as degraded at 65.188% next to es-01 and es-02 at about 46%](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-java-heap-usage-analysis.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=4adbed08921da975d9f321deddc2bb96)

Analysis of Java Heap Usage - GermainUX

### Investigate a Node That Leaves the Cluster

When the cluster turns yellow, open the **Elasticsearch Cluster Node Count** KPI. The top row shows how many nodes the cluster had on average, against the previous days, and the hourly trend shows when a node left and came back.

In the example below, **es-03** drops out of the cluster after the heap pressure above. The cluster is yellow while Elasticsearch rebuilds the replicas of **es-03** on **es-01** and **es-02**. Between 07:00 and 12:00 UTC on 11 November, the cluster averages 2.850 nodes against 3.000 on the previous days, down 5.0%. The hourly trend dips and recovers when **es-03** rejoins, about 45 minutes later.

![Analysis screen for the Elasticsearch Cluster Node Count KPI, with an average of 2.850 nodes against 3.000 on the previous days, and a dip in the hourly trend and the relationships chart](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-node-count-analysis.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=18371d8bbdfb007d2046740b209b9733)

Analysis of Elasticsearch Cluster Node Count - GermainUX

### Investigate a Disk Watermark

When a node's disk passes the low disk watermark, **Elasticsearch Errors** records an event, and Elasticsearch stops placing new shards on that node. Open the **Disk Usage** KPI to see which node is filling up and how fast. Click **Show all segments** to compare the nodes under **By System Name**.

In the example below, log indices that are never deleted fill the disk of **es-02** past the `85%` low watermark. Between 05:00 and 16:00 UTC on 13 November, the average disk usage is 64.408%, up 6.8% on the previous days. Under **By System Name**, **es-02** averages 73.550%, up 18.0%, while **es-01** and **es-03** stay near 59% and 61%. The line of **es-02** climbs for several hours, and drops once the old indices are deleted. The **Disk Usage** SLA of `75%` warns you before the node reaches the watermark.

![Analysis screen for the Disk Usage KPI filtered to the Elasticsearch application, with an average of 64.408% up 6.8% on the previous days, and es-02 at 73.550% up 18.0% under By System Name, next to es-01 and es-03](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-disk-usage-analysis.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=bbf92beb457d4bd5cbfa1e791278cf99)

Analysis of Disk Usage - GermainUX

### Investigate a Red Cluster After a Restart

When the cluster turns red, some primary shards have no copy on any node. Open the **Elasticsearch Cluster Health** KPI, as in the analysis under [Dashboards and Reports](#dashboards-and-reports), to see how long the cluster stayed red. Then open **Elasticsearch Active Shards** to see how many shards were missing, and whether all of them are active again.

In the example below, **es-02** restarts for patching at night. The `audit` index has two primary shards and no replicas, both on **es-02**, so the cluster stays red until **es-02** is back. Between 22:00 UTC on 13 November and 04:00 UTC on 14 November, the cluster averages 86.125 active shards, against 88.000 on the previous days, down 2.1%. The hourly trend dips while **es-02** is away, and returns to all shards active once it rejoins.

![Analysis screen for the Elasticsearch Active Shards KPI, with an average of 86.125 active shards against 88.000 on the previous days, and a dip in the hourly trend and the relationships chart](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-active-shards-analysis.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=8878ee0fc1519f72d9008840a836190d)

Analysis of Elasticsearch Active Shards - GermainUX

### Investigate an Unreachable Cluster Endpoint

When the Elasticsearch charts stop updating, open the **HTTP Availability** KPI. It shows whether the cluster's REST endpoint answered each poll. While the endpoint doesn't answer, the other Elasticsearch KPIs get no values.

In the example below, the load balancer `search-prod-lb` doesn't answer for 25 minutes during network maintenance. Between 06:00 and 11:00 UTC on 15 November, availability averages 91.667%, against 100.000% on the previous days, down 8.3%. The breakdown marks `search-prod-lb` as degraded, at 0.000% for its 5 failed polls. The **HTTP Availability** SLA of `95%` raises an alert when the endpoint stops answering.

![Analysis screen for the HTTP Availability KPI filtered to the Elasticsearch application, with an average of 91.667% against 100.000% on the previous days, a dip in the relationships chart, and search-prod-lb marked as degraded at 0.000%](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-http-availability-analysis.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=6f8dbda4e485789fb057b083d8df6987)

Analysis of HTTP Availability - GermainUX

### SLAs and Alerts

The cluster KPIs come with default [SLAs](https://docs.germainux.com/analytics/slas.md), listed under [KPIs](#kpis). Get an [alert](https://docs.germainux.com/automation/notifications.md) by email, SMS, Slack or Teams when the cluster turns yellow or red, when a node's heap or disk stays over its limit, or when the endpoint stops answering.

### Automation

When an SLA is breached, an [automated action](https://docs.germainux.com/automation/automated-actions.md) can run a diagnostic script, call an HTTP endpoint or open an incident.

### Correlation with Other Technologies

[Correlate](https://docs.germainux.com/analytics/correlation-and-tracing.md) the cluster's health with the applications that use it, such as a [Java](https://docs.germainux.com/integrations/java-monitoring.md) service whose searches slow down while a node is short of heap.

## How It Works

Elasticsearch REST API
Elasticsearch HTTP Monitor
Elasticsearch rules
KPIs
Alerts

| Component | What it does |
| - | - |
| **Elasticsearch HTTP Monitor** | Runs on a GermainUX Engine and calls the cluster's REST API on a schedule: `_cluster/settings`, `_cluster/health`, `_nodes/stats`, `_rollup/job/_all` and `_cat/shards` |
| **Elasticsearch rules** | Turn each response into KPI values. Cluster values are named after the cluster, and node values use the node as the system name, so SLAs apply to each node separately. |
| **Elasticsearch Query Monitor** | Runs your Elasticsearch SQL or Query DSL queries on a schedule. A [rule](https://docs.germainux.com/automation/rules.md) that you assign turns each returned row or hit into a KPI value. |

One HTTP monitor covers the whole cluster, because the cluster APIs answer for every node. Point it at any node, or at the load balancer in front of the cluster.

## Before You Start

* A GermainUX Engine can reach the cluster's REST port, usually `9200`. See [GermainUX Engine](https://docs.germainux.com/deploy/germain-ux-engine.md).
* If the cluster has security turned on, an Elasticsearch user for GermainUX. For cluster monitoring, it needs the `monitor` cluster privilege and the `monitor` index privilege. For the Query Monitor, it also needs `read` on the indices you query.
* That user is saved as credentials in **Settings > Data Sources > Credentials**. See [Data Sources](https://docs.germainux.com/management/data-source.md).

## Deploy

> [tab] Cluster Monitoring
>
> **1. Open the HTTP Scenario wizard**
>
> In the left menu, go to **Settings > Wizards**. On the **HTTP Scenario Component Deployment** card, click **Add**.
>
> **2. Choose the cluster endpoint**
>
> On the **Server** step, fill in the fields:
>
> | Field | Value |
> | - | - |
> | **URI** | A URI data source for the cluster's REST endpoint, for example `http://es-01:9200`. Select a saved one, or create it from the field. |
> | **Credentials** | The Elasticsearch user, if the cluster has security turned on |
> | **Application Name** | `Elasticsearch`, so the node charts of the **Elasticsearch Overview** dashboard show this cluster |
>
> **3. Select the Elasticsearch HTTP Monitor**
>
> On the **Component** step, set **Component Type** to `Elasticsearch HTTP Monitor`.
>
> If you selected credentials, click **Show Advanced**, and set **Auth Policy** to `Basic`. Without an auth policy, GermainUX doesn't send the credentials to the cluster.
>
> ![HTTP Scenario Component Deployment wizard on the Component step, with Elasticsearch HTTP Monitor selected as the Component Type](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/http-scenario-deployment-elasticsearch.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=7b7febc6ed4a0366db8df5a07e8d1c19)
>
> HTTP Scenario Component Deployment Wizard for Elasticsearch - GermainUX
>
> **4. Deploy the monitor**
>
> On the **Deployment** step, fill in the fields, and finish the wizard:
>
> | Field | Value |
> | - | - |
> | **Monitor Name** | A name you recognize, for example `Elasticsearch HTTP Monitor - search-prod` |
> | **Monitoring Node** | The Monitoring Node and Engine that can reach the cluster |
> | **Schedule on which to run this component** | How often to poll the cluster. The default runs once an hour. A shorter interval, such as every 5 minutes (`0 */5 * * * *`), catches short yellow or red periods. |

> [tab] Query Monitor
>
> The Query Monitor needs the cluster as a database data source and a component that holds your queries. Create the component first, and then deploy it.
>
> **1. Add the cluster as a database**
>
> In the left menu, go to **Settings > Data Sources**. On the **Databases** tab, add a database with **Vendor** set to `Elasticsearch`, the **Host** of a node or load balancer, and **Port** `9200`.
>
> For a cluster that uses HTTPS, select **Configure with URL?**, and enter the cluster URL in **JDBC URL**, for example `https://es-01:9200`.
>
> **2. Create the query component**
>
> In the left menu, go to **Settings > Wizards**. On the **Elasticsearch Query Monitor Component** card, click **Add**.
>
> * **Component**: enter a **Component Name**, for example `Orders Index Checks`.
> * **Validation**: select the Elasticsearch **Database** and the **Credentials** to test your queries with.
>
> **3. Write the query**
>
> On the **Query** step, enter a **Query Name**, and choose the **Format**:
>
> | Format | What to enter |
> | - | - |
> | `SQL` | An Elasticsearch SQL query in **SQL** |
> | `JSON` | The index to search in **Index** (leave it empty to search all indices), and a Query DSL request body in **JSON** |
>
> For example, this SQL query counts the failed orders in an `orders` index:
>
> ```sql theme={null}
> SELECT COUNT(*) AS failed_orders FROM "orders" WHERE status = 'FAILED'
> ```
>
> Click **Validate Query** to run it against the database, and then finish the wizard.
>
> ![Elasticsearch Query Monitor Component wizard on the Query step, with a query name, the SQL format and a count query filled in](https://mintcdn.com/germain-ux/cSCoMMl7KW61OHIM/images/main/elasticsearch-monitoring/elasticsearch-query-monitor-query-step.png?fit=max\&auto=format\&n=cSCoMMl7KW61OHIM\&q=85\&s=a1724b76586a868a6e5ffbec3ce4bfe9)
>
> Query Step of the Elasticsearch Query Monitor Component Wizard - GermainUX
>
> **4. Assign a rule to the component**
>
> The query component turns results into KPI values through a rule that you write. In the left menu, go to **Settings > System > Engine Settings > Component Types**, and open your component. Select **Use Custom Rule?**, and choose your rule in **Custom Rule Name**. See [Create a Rule](https://docs.germainux.com/automation/rules/create-a-rule-for-advanced-automation.md).
>
> **5. Deploy the query monitor**
>
> On the **Elasticsearch Query Monitor Deployment** card, click **Add**, fill in the fields, and finish the wizard:
>
> | Field | Value |
> | - | - |
> | **Database** | The Elasticsearch database from the first step |
> | **Credentials** | The Elasticsearch user |
> | **Application Name** | The application the query results belong to |
> | **Component** | The query component you created |
> | **Monitor Name** | For example `Elasticsearch Query Monitor - orders` |
> | **Monitoring Node** | The Monitoring Node and Engine that can reach the cluster |
> | **Schedule on which to run this component** | How often to run the queries. The default runs once an hour. |

## Validate

**1. Check the cluster from the Engine**

On the Engine server, call the cluster health API:

```bash theme={null}
curl -s -u ES_USER "http://ES_HOST:9200/_cluster/health?pretty"
```

Replace `ES_HOST` with the host in the URI data source, and `ES_USER` with the Elasticsearch user. Leave out `-u ES_USER` if the cluster has security turned off. The response shows the cluster `status` that GermainUX reports.

**2. Find the cluster in GermainUX**

After the next scheduled run, open the **Elasticsearch Overview** dashboard. **Cluster Health** shows the cluster's status, and the node charts show each node. For the Query Monitor, open the KPI that your rule writes to.

## Troubleshooting

> [accordion] No Elasticsearch data appears
>
> Check that the Engine reaches the URI with the `curl` command under [Validate](#validate), that the **Component Type** is `Elasticsearch HTTP Monitor`, and that the scheduled time has passed. With the default schedule, the first poll can take up to an hour.

> [accordion] The monitor gets 401 Unauthorized
>
> Select the Elasticsearch user in **Credentials**, and set **Auth Policy** to `Basic` under **Show Advanced** on the **Component** step. Check that the user has the privileges listed under [Before You Start](#before-you-start).

> [accordion] The node charts on the dashboard are empty
>
> The node charts of **Elasticsearch Overview** show only monitors whose application is `Elasticsearch`. Set **Application Name** to `Elasticsearch` on the HTTP monitor.

> [accordion] The Query Monitor runs, but no KPI values appear
>
> The query component needs a rule that turns its results into KPI values. Assign one with **Use Custom Rule?** on the component, as in the Query Monitor steps under [Deploy](#deploy).

> [accordion] The Query Monitor can't connect to an HTTPS cluster
>
> Without a URL, the Query Monitor connects over plain HTTP to the database's host and port. Select **Configure with URL?** on the database data source, and enter the `https://` URL in **JDBC URL**.

## Security and Privacy

Elasticsearch monitoring data includes:

* **Infrastructure identifiers**: the cluster name, and the name and IP address of each node
* **Query results**: the Query Monitor stores the fields that your queries return and your rule maps. Indexed documents can hold personal data or free text, so treat these results as possibly containing personal or customer data.

To mask, anonymize or exclude this data, see [Data Privacy](https://docs.germainux.com/management/compliance-privacy-and-security/data-privacy.md).

## KPIs

| KPI | What it holds | Default SLA |
| - | - | - |
| **Elasticsearch Cluster Health** | The cluster status: `100` for green, `50` for yellow, `0` for red | Orange when the cluster is yellow, red when it's red |
| **Elasticsearch Cluster Node Count** | The number of nodes in the cluster | Breached below `1` |
| **Elasticsearch Active Shards** | The number of active shards in the cluster | Breached below `1` |
| **Elasticsearch Shard Count** | The number of shards in the cluster, and on each node | Breached below `1` |
| **Elasticsearch Read Ops** | Read operations of each node since it started | — |
| **Elasticsearch Write Ops** | Write operations of each node since it started | — |
| **Elasticsearch Rollup Job** | The documents each rollup job has processed | — |
| **Elasticsearch Errors** | An event when a node's disk passes the low disk watermark | Breached on every event |

The HTTP monitor also feeds these shared KPIs, which GermainUX uses for other technologies too:

| KPI | What it holds | Default SLA |
| - | - | - |
| **HTTP Availability** | Whether the cluster's REST endpoint answers | Breached below `95%` |
| **CPU Usage**, **Memory Usage**, **Disk Usage**, **Swap Usage** | Resource usage of each node, in percent | CPU `90%`, memory `80%`, disk `75%` |
| **Java Heap Usage** | JVM heap usage of each node, in percent | Breached above `75%` |
| **Java Thread Count** | JVM threads of each node | Breached above `500` |
| **OS Process** | CPU and memory of the Elasticsearch process on each node | — |

Adjust the thresholds to your cluster. See [KPIs](https://docs.germainux.com/analytics/kpis.md) and [SLAs](https://docs.germainux.com/analytics/slas.md).

## Related Documentation

> [card] [Synthetic HTTP Execution](https://docs.germainux.com/automation/synthetic-http-execution.md)
>
> How HTTP monitors run on a schedule

> [card] [Create a Rule](https://docs.germainux.com/automation/rules/create-a-rule-for-advanced-automation.md)
>
> Turn query results into KPIs

> [card] [Automation](https://docs.germainux.com/automation.md)
>
> More Automation Features

## Get Help

The GermainUX team can help you write Query Monitor rules for your indices, and tune the SLAs to your cluster.

> [card] [Contact GermainUX Support](https://docs.germainux.com/getting-started/support.md)
>
> The GermainUX team helps you get up and running faster, empower your team, turn data into actionable insights, and optimize performance to deliver measurable business outcomes.

> [card] Can't Find What You Need?
>
> Search all GermainUX docs with **Ctrl + K**, or use the feedback buttons at the bottom of this page to tell us what is missing.

The content of this document is proprietary to GERMAIN SOFTWARE.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.