> ## Documentation Index
> Fetch the complete documentation index at: https://docs.germainux.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sentinel: GermainUX Self-Monitoring

> Sentinel independently monitors the health, availability, and performance of the GermainUX platform.

Service
`Enterprise`

Availability
2023-01-01

> [card] [Email reports](#email-reports)
>
> Consolidated status email

> [card] [Service-availability checks](#service-availability-checks)
>
> Watch critical OS services

> [card] [Messaging checks](#messaging-checks)
>
> Monitor queue backlog and consumers

> [card] [Troubleshooting](#troubleshooting)
>
> Fix Sentinel report issues

**Sentinel: GermainUX Self-Monitoring**

Sentinel independently monitors the health, availability, and performance of the GermainUX platform. It detects failures or degradation across the infrastructure, services, logs, and HTTP endpoints required by GermainUX, then summarizes the findings in a single email report.

Because Sentinel operates separately from the GermainUX components it monitors, it can report a GermainUX outage even when the main platform is unavailable.

## What Sentinel monitors

Sentinel can perform the following checks:

| Area | Checks |
| - | - |
| Infrastructure | Availability of required infrastructure services, such as ZooKeeper and the configured messaging layer |
| Messaging | Broker availability, resource utilization, queue depth, consumer count, and message-processing activity |
| Operating-system services | Service availability, process IDs, CPU usage, and memory usage |
| Log files | File availability, last-modified time, recent errors, and warnings |
| HTTP endpoints | Availability, response status, and response time |
| GermainUX Services | Availability and log activity for Action, Aggregation, Analytics, Session Tracking, Storage, and related services |
| Germain Enterprise | Availability of configuration, ingestion, query, and other Enterprise endpoints |

Legacy Sentinel report examples may reference ActiveMQ. GermainUX 2024.2 and later uses Kafka. Confirm that the Sentinel package and configuration used by your release contain the appropriate messaging checks.

## Health status

Sentinel classifies each check using the following status levels:

| Status | Meaning |
| - | - |
| **Red** | The component is unavailable, failing, or not functioning correctly |
| **Orange** | The component is available but slow, degraded, or generating errors |
| **Green** | The component is available and performing within the configured thresholds |

The thresholds that produce each status depend on the Sentinel configuration.

## Email reports

Sentinel consolidates all findings into a single email report. Report-delivery conditions are configurable, allowing reports to be sent:

* After every Sentinel execution
* Only when a warning or failure is detected
* When a component’s status changes
* According to another environment-specific condition

A report should make it possible to identify:

* Which component is affected
* Which check failed
* Whether the issue is an outage or degradation
* The evidence that produced the status
* The host or endpoint involved
* When the underlying service or log was last active

- After every Sentinel execution
- Only when a warning or failure is detected
- When a component’s status changes
- According to another environment-specific condition

* Which component is affected
* Which check failed
* Whether the issue is an outage or degradation
* The evidence that produced the status
* The host or endpoint involved
* When the underlying service or log was last active

The exact report format and available checks depend on the Sentinel version and configuration.

## Service-availability checks

Sentinel can monitor a configurable list of operating-system services.

A service check may report:

```text theme={null}
Service: GermainStorageServices
Check: AvailabilityCheck
Status: Running
PID: 1648
```

Configure checks for all GermainUX processes required by the environment, including applicable:

* Germain Enterprise services
* Action Services
* Aggregation Services
* Analytics Services
* Session Tracking Services
* Storage Services

> [accordion] Show all 9
>
> * Real-Time Metrics Services
> * Germain Engine Managers
> * Infrastructure services

* Germain Enterprise services
* Action Services
* Aggregation Services
* Analytics Services
* Session Tracking Services
* Storage Services

> [accordion] Show all 9
>
> * Real-Time Metrics Services
> * Germain Engine Managers
> * Infrastructure services

The exact service names depend on the operating system and deployment configuration.

## Log-activity checks

Sentinel can monitor local or remote log files and report:

* Log path
* Last-modified timestamp
* Time since the last update
* Number of recent errors
* Number of recent warnings
* Whether configured thresholds were exceeded

Example:

```text theme={null}
Service: GermainAnalyticsServices
Check: LogActivity
Path: E:\Software\GermainAPMService\var\logs\analytics-services.log
Modified: 0.11 minutes ago
Errors: 0
```

A stale log does not always indicate a failure. Some services write only when they process activity. Set inactivity thresholds according to the expected workload of each component.

When monitoring remote Windows paths, the Sentinel account must be able to access the share, for example:

```text theme={null}
\\<server>\e$\Software\GermainAPMService\var\logs\
```

Administrative shares may be restricted by security policies. Prefer a dedicated, read-only share where required.

## HTTP endpoint checks

Sentinel can call configured HTTP endpoints and report their response status.

Example:

```text theme={null}
Service: GermainAPMQueryServices
Check: EndpointAvailability
REST endpoint response code: 200
```

Configure endpoint checks for the critical GermainUX services that must remain reachable.

A successful HTTP connection does not always prove that the service is fully operational. Where supported, configure checks against endpoints that validate required downstream dependencies.

## Messaging checks

Sentinel can monitor the messaging layer used to transfer work between GermainUX components.

Typical metrics include:

* Broker availability
* Queue or topic backlog
* Consumer count
* Messages received
* Messages processed
* Memory utilization
* Persistent-store utilization
* Temporary-store utilization

A legacy ActiveMQ queue report may resemble:

```text theme={null}
Queue: apm.storage
QueueSize: 0
ConsumerCount: 2
EnqueueCount: 7480961
DequeueCount: 7480961
```

Interpret these values together:

* A growing queue indicates that consumers may not be processing messages quickly enough.
* A consumer count of zero may indicate that the corresponding GermainUX Service is unavailable.
* Increasing enqueue counts with unchanged dequeue counts indicate stalled processing.
* High memory or store utilization can prevent the broker from accepting new messages.

For Kafka-based deployments, use the equivalent broker, topic, partition, consumer-group, and lag checks available in the applicable Sentinel release.

## Recommended configuration

At minimum, configure Sentinel to monitor:

* ZooKeeper availability
* Kafka availability and consumer health—or ActiveMQ for legacy releases
* Hazelcast availability
* Germain Enterprise HTTP endpoints
* Every required GermainUX operating-system service
* GermainUX Service logs

> [accordion] Show all 10
>
> * Tomcat and Enterprise logs
> * Germain Engine Manager processes and logs
> * Datastore availability
> * Disk utilization on hosts containing logs, queues, or persistent data

Use different thresholds for development and production. Production thresholds should reflect normal workload, maintenance windows, and agreed service levels.

## Deployment considerations

Sentinel should run independently from the GermainUX components it monitors whenever possible.

If Sentinel runs on the same server as Germain Enterprise, a server-level outage may prevent both GermainUX and Sentinel from operating or sending an alert.

### Host capability checklist

Confirm that the Sentinel host can:

| | Requirement |
| - | - |
| 1 | Resolve every monitored hostname |
| 2 | Reach the required service ports and HTTP endpoints |
| 3 | Read the configured log files |
| 4 | Query the required operating-system services |
| 5 | Connect to the SMTP server |
| 6 | Access messaging and infrastructure metrics |
| 7 | Write its own logs and temporary files |

Use a dedicated service account with the minimum permissions required to perform these checks.

## Validate Sentinel

After configuring Sentinel:

**1. Run it manually**

**2. Confirm that every expected component appears in the report**

**3. Verify that healthy components report green**

**4. Stop or simulate the failure of a non-production test service**

**5. Confirm that Sentinel detects the failure**

**6. Verify that the report is delivered to the intended recipients**

**7. Restore the test service**

**8. Confirm that Sentinel reports recovery**

**9. Review Sentinel’s own logs for errors**

Also verify warning behavior by testing a non-critical threshold, such as a controlled stale-log or slow-endpoint condition.

## Avoid false alerts

To reduce false or misleading reports:

* Configure realistic response-time thresholds.
* Account for services that write logs infrequently.
* Exclude planned maintenance periods where supported.
* Monitor the correct process when multiple instances run on one host.
* Use stable endpoint URLs.
* Confirm that remote log shares remain available.
* Review thresholds after major changes in monitoring volume.
* Update Sentinel whenever GermainUX services, hosts, paths, or ports change.

## Troubleshooting

### A service is reported as unavailable but is running

Confirm that:

* Sentinel uses the correct operating-system service name.
* The Sentinel account can query the service.
* Sentinel is checking the correct host.
* Multiple services do not share a confusing display name.
* Hostname resolution and remote-management access work.

### A log is reported as stale

Verify:

* The configured path is correct.
* The service is expected to write during the monitored period.
* The Sentinel account can read the file.
* The remote share is accessible.
* The inactivity threshold is appropriate.
* Log rotation did not move activity to a new filename.

### An endpoint check fails

Confirm:

* The URL and port are correct.
* DNS resolves from the Sentinel host.
* Firewalls permit the request.
* TLS certificates are valid and trusted.
* Authentication is configured when required.
* The target service and its dependencies are running.

### The report is not delivered

Check:

* SMTP hostname and port
* Authentication credentials
* TLS settings
* Sender and recipient addresses
* Firewall access
* Report-delivery conditions
* Sentinel’s own logs

## Operational ownership

Assign responsibility for:

* Reviewing Sentinel reports
* Responding to red and orange conditions
* Maintaining recipient lists
* Updating checks after infrastructure changes
* Testing Sentinel after GermainUX upgrades
* Periodically confirming that failure alerts are delivered

A self-monitoring system is useful only if its own execution and notification delivery are tested regularly.

## Get Help

> [card] [Contact GermainUX Support](https://docs.germainux.com/main/support.md)
>
> The GermainUX team helps you get up and running faster, empower your team, turn data into actionable insights, and optimize performance to deliver measurable business outcomes.

> [card] Can't find what you need?
>
> Search all GermainUX docs with **Ctrl + K**, or use the feedback buttons at the bottom of this page to tell us what is missing.

The content of this document is proprietary to GERMAIN SOFTWARE.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.