Capacity issue with an Engine? | Germain Documentation

⚙️ Resolve GermainUX Engine Capacity Issues

A GermainUX Engine may slow down, miss scheduled executions, or restart when it or its host does not have enough CPU or memory.

The correct solution depends on whether the constraint affects the Engine’s JVM or the entire host.

🔍 Symptoms

Common symptoms include:

Symptom
Slow monitoring-component execution
Delayed telemetry
Missed or overlapping execution schedules
High CPU utilization
High JVM heap utilization
Frequent garbage collection
`OutOfMemoryError`
Unexpected Engine restarts
Increasing local backlog
Host paging or swapping

🧭 1. Identify the constrained resource

Check both the Engine and its host:

Resource
Engine JVM heap usage
Engine and Engine Manager CPU usage
Host physical-memory utilization
Host CPU utilization
Swap or paging activity
Available disk space
Number of Engines on the host
Number and frequency of assigned components

Review the Engine and Engine Manager logs for:

OutOfMemoryError
GC overhead limit exceeded
Java heap space
Unable to create new native thread

Also look for components whose execution time exceeds their configured frequency.

📚 2. If the Engine lacks JVM memory

If the Engine is approaching its maximum heap but the host still has sufficient free memory, increase its -Xmx value.

In the GermainUX Workspace:

Open Germain > State.
Select the Engines view.
Find and select the affected Engine.
Update its maximum heap setting:
```
-Xmx<size>
```
Save the configuration.
Select Restart this Engine.
Verify the new value in the process command line or logs.
Monitor the Engine and host after the change.

Examples:

-Xmx2g
-Xmx4g

Do not allocate all host memory to the Engine. Reserve capacity for:

Reserved for
Operating system
Engine Manager
Other Engines
Native JVM memory
Drivers and libraries
Monitoring tools
Other applications on the host

💻 3. If the host lacks CPU or memory

Increasing the Engine heap will not resolve host-level resource exhaustion.

Use one or more of the following options:

🚚 Move an Engine

Relocate the complete Engine instance to another Engine host with available capacity.

This is appropriate when its components belong together or share local access requirements.

🔁 Move selected components

Reassign resource-intensive monitoring or automation components to an Engine on another host.

This is appropriate when only part of the Engine workload causes the constraint.

➕ Add Engine hosts

Deploy additional Engine hosts and distribute Engines or components across them.

This is the preferred approach when monitoring volume or component count has grown permanently.

✂️ Reduce unnecessary workload

Review:

What to review
Component execution frequency
Duplicate monitoring
Obsolete components
Queries returning excessive data
Log scans covering unnecessarily large files
Synthetic scenarios running too frequently
Scripts or actions with long execution times
Components repeatedly failing and retrying

Do not reduce monitoring frequency without considering the effect on detection time, alerting, SLAs, and automation.

💡 4. Identify the source of the load

Determine whether capacity pressure is:

Pattern
Continuous
Limited to certain times
Associated with one component
Associated with a recent configuration change
Caused by increasing monitored-data volume
Caused by multiple Engines competing on the same host

Temporarily disabling a suspected component in a non-production environment can help confirm its effect. Do not disable critical production monitoring without approval.

⚖️ 5. Balance Engines across hosts

When redistributing the workload, consider:

Consideration
CPU and memory requirements
Access to monitored systems
Network latency
Firewall rules
JDBC drivers
Local file access
Service-account permissions
Proxy requirements
Automation security
Maintenance and failure domains

Components that read local logs or operating-system resources generally need an Engine with access to the target host or file share.

✅ Validate the solution

After increasing memory or redistributing the workload, confirm that:

Check
The Engine starts successfully.
Its heartbeat appears in Germain > State.
All required components are running.
Execution delays decrease.
Telemetry reaches Germain Enterprise on time.
JVM heap utilization remains stable.
Garbage-collection activity returns to an acceptable level.
Host CPU and memory utilization remain within safe limits.
No new capacity errors appear in the logs.
Alerts and automations continue to operate as expected.

Monitor the environment through at least one representative peak workload period.

📊 Capacity-planning recommendations

To reduce future capacity problems:

Recommendation
Establish baseline CPU and memory usage.
Monitor Engine and host utilization.
Track component count and execution frequency.
Review capacity after adding applications or monitored users.
Avoid concentrating all critical Engines on one host.
Maintain spare capacity for workload spikes.
Test large configuration changes before broad deployment.
Alert on sustained CPU, memory, disk, and execution-delay thresholds.

ℹ️ Information to collect for Support

If the problem persists, provide:

Item
GermainUX and Engine versions
Engine and node names
Engine Manager and Engine logs
JVM options
Engine component list
Component execution frequencies
Host CPU and memory utilization
JVM heap utilization
Swap or paging activity
Disk utilization
Time periods when degradation occurs
Recent workload or configuration changes

Remove credentials and other secrets before sharing logs or configuration.

Service: Enterprise

Feature Availability: 2024-01-01