Sentinel: GermainUX Self-Monitoring | Germain Documentation

🛰 Sentinel: GermainUX Self-Monitoring

Sentinel independently monitors the health, availability, and performance of the GermainUX platform. It detects failures or degradation across the infrastructure, services, logs, and HTTP endpoints required by GermainUX, then summarizes the findings in a single email report.

Because Sentinel operates separately from the GermainUX components it monitors, it can report a GermainUX outage even when the main platform is unavailable.

🔍 What Sentinel monitors

Sentinel can perform the following checks:

Area	Checks
Infrastructure	Availability of required infrastructure services, such as ZooKeeper and the configured messaging layer
Messaging	Broker availability, resource utilization, queue depth, consumer count, and message-processing activity
Operating-system services	Service availability, process IDs, CPU usage, and memory usage
Log files	File availability, last-modified time, recent errors, and warnings
HTTP endpoints	Availability, response status, and response time
GermainUX Services	Availability and log activity for Action, Aggregation, Analytics, Session Tracking, Storage, and related services
Germain Enterprise	Availability of configuration, ingestion, query, and other Enterprise endpoints

Legacy Sentinel report examples may reference ActiveMQ. GermainUX 2024.2 and later uses Kafka. Confirm that the Sentinel package and configuration used by your release contain the appropriate messaging checks.

💓 Health status

Sentinel classifies each check using the following status levels:

Status	Meaning
Red	The component is unavailable, failing, or not functioning correctly
Orange	The component is available but slow, degraded, or generating errors
Green	The component is available and performing within the configured thresholds

The thresholds that produce each status depend on the Sentinel configuration.

📧 Email reports

Sentinel consolidates all findings into a single email report. Report-delivery conditions are configurable, allowing reports to be sent:

Condition
After every Sentinel execution
Only when a warning or failure is detected
When a component’s status changes
According to another environment-specific condition

A report should make it possible to identify:

Item
Which component is affected
Which check failed
Whether the issue is an outage or degradation
The evidence that produced the status
The host or endpoint involved
When the underlying service or log was last active

Condition
After every Sentinel execution
Only when a warning or failure is detected
When a component’s status changes
According to another environment-specific condition

Item
Which component is affected
Which check failed
Whether the issue is an outage or degradation
The evidence that produced the status
The host or endpoint involved
When the underlying service or log was last active

The exact report format and available checks depend on the Sentinel version and configuration.

⚙️ Service-availability checks

Sentinel can monitor a configurable list of operating-system services.

A service check may report:

Service: GermainStorageServices
Check: AvailabilityCheck
Status: Running
PID: 1648

Configure checks for all GermainUX processes required by the environment, including applicable:

Process/Service
Germain Enterprise services
Action Services
Aggregation Services
Analytics Services
Session Tracking Services
Storage Services
Real-Time Metrics Services
Germain Engine Managers
Infrastructure services

Process/Service
Germain Enterprise services
Action Services
Aggregation Services
Analytics Services
Session Tracking Services
Storage Services
Real-Time Metrics Services
Germain Engine Managers
Infrastructure services

The exact service names depend on the operating system and deployment configuration.

📃 Log-activity checks

Sentinel can monitor local or remote log files and report:

Metric
Log path
Last-modified timestamp
Time since the last update
Number of recent errors
Number of recent warnings
Whether configured thresholds were exceeded

Example:

Service: GermainAnalyticsServices
Check: LogActivity
Path: E:\Software\GermainAPMService\var\logs\analytics-services.log
Modified: 0.11 minutes ago
Errors: 0

A stale log does not always indicate a failure. Some services write only when they process activity. Set inactivity thresholds according to the expected workload of each component.

When monitoring remote Windows paths, the Sentinel account must be able to access the share, for example:

\\<server>\e$\Software\GermainAPMService\var\logs\

Administrative shares may be restricted by security policies. Prefer a dedicated, read-only share where required.

🌐 HTTP endpoint checks

Sentinel can call configured HTTP endpoints and report their response status.

Example:

Service: GermainAPMQueryServices
Check: EndpointAvailability
REST endpoint response code: 200

Configure endpoint checks for the critical GermainUX services that must remain reachable.

A successful HTTP connection does not always prove that the service is fully operational. Where supported, configure checks against endpoints that validate required downstream dependencies.

💬 Messaging checks

Sentinel can monitor the messaging layer used to transfer work between GermainUX components.

Typical metrics include:

Metric
Broker availability
Queue or topic backlog
Consumer count
Messages received
Messages processed
Memory utilization
Persistent-store utilization
Temporary-store utilization

A legacy ActiveMQ queue report may resemble:

Queue: apm.storage
QueueSize: 0
ConsumerCount: 2
EnqueueCount: 7480961
DequeueCount: 7480961

Interpret these values together:

Observation
A growing queue indicates that consumers may not be processing messages quickly enough.
A consumer count of zero may indicate that the corresponding GermainUX Service is unavailable.
Increasing enqueue counts with unchanged dequeue counts indicate stalled processing.
High memory or store utilization can prevent the broker from accepting new messages.

For Kafka-based deployments, use the equivalent broker, topic, partition, consumer-group, and lag checks available in the applicable Sentinel release.

💡 Recommended configuration

At minimum, configure Sentinel to monitor:

Item
ZooKeeper availability
Kafka availability and consumer health—or ActiveMQ for legacy releases
Hazelcast availability
Germain Enterprise HTTP endpoints
Every required GermainUX operating-system service
GermainUX Service logs
Tomcat and Enterprise logs
Germain Engine Manager processes and logs
Datastore availability
Disk utilization on hosts containing logs, queues, or persistent data

Use different thresholds for development and production. Production thresholds should reflect normal workload, maintenance windows, and agreed service levels.

🚀 Deployment considerations

Sentinel should run independently from the GermainUX components it monitors whenever possible.

If Sentinel runs on the same server as Germain Enterprise, a server-level outage may prevent both GermainUX and Sentinel from operating or sending an alert.

✅ Host capability checklist

Confirm that the Sentinel host can:

	Requirement
1	Resolve every monitored hostname
2	Reach the required service ports and HTTP endpoints
3	Read the configured log files
4	Query the required operating-system services
5	Connect to the SMTP server
6	Access messaging and infrastructure metrics
7	Write its own logs and temporary files

Use a dedicated service account with the minimum permissions required to perform these checks.

🏳️ Validate Sentinel

After configuring Sentinel:

Run it manually.
Confirm that every expected component appears in the report.
Verify that healthy components report green.
Stop or simulate the failure of a non-production test service.
Confirm that Sentinel detects the failure.
Verify that the report is delivered to the intended recipients.
Restore the test service.
Confirm that Sentinel reports recovery.
Review Sentinel’s own logs for errors.

Also verify warning behavior by testing a non-critical threshold, such as a controlled stale-log or slow-endpoint condition.

🔕 Avoid false alerts

To reduce false or misleading reports:

Recommendation
Configure realistic response-time thresholds.
Account for services that write logs infrequently.
Exclude planned maintenance periods where supported.
Monitor the correct process when multiple instances run on one host.
Use stable endpoint URLs.
Confirm that remote log shares remain available.
Review thresholds after major changes in monitoring volume.
Update Sentinel whenever GermainUX services, hosts, paths, or ports change.

🔧 Troubleshooting

ℹ️ A service is reported as unavailable but is running

Confirm that:

Check
Sentinel uses the correct operating-system service name.
The Sentinel account can query the service.
Sentinel is checking the correct host.
Multiple services do not share a confusing display name.
Hostname resolution and remote-management access work.

📁 A log is reported as stale

Verify:

Check
The configured path is correct.
The service is expected to write during the monitored period.
The Sentinel account can read the file.
The remote share is accessible.
The inactivity threshold is appropriate.
Log rotation did not move activity to a new filename.

🔗 An endpoint check fails

Confirm:

Check
The URL and port are correct.
DNS resolves from the Sentinel host.
Firewalls permit the request.
TLS certificates are valid and trusted.
Authentication is configured when required.
The target service and its dependencies are running.

📬 The report is not delivered

Check:

Check
SMTP hostname and port
Authentication credentials
TLS settings
Sender and recipient addresses
Firewall access
Report-delivery conditions
Sentinel’s own logs

🤝 Operational ownership

Assign responsibility for:

Responsibility
Reviewing Sentinel reports
Responding to red and orange conditions
Maintaining recipient lists
Updating checks after infrastructure changes
Testing Sentinel after GermainUX upgrades
Periodically confirming that failure alerts are delivered

A self-monitoring system is useful only if its own execution and notification delivery are tested regularly.

Service: Enterprise

Feature Availability: 2023-01-01