Maintenance - Self-monitoring script
Description
We have completed v2 of the self-monitoring script that monitors and ensures high availability and performance of Germain. That script performs the following:
Check Availability of Germain Services (ActiveMQ, ApacheSolr, Action, Aggregation, Analytics, Config, Engines, Ingestion Services, Query Services, Engine Manager, Session Tracking and Storage Services)
Check Updates and Errors in Germain’s Logs
Check Availability of Germain’s REST Api
Check Availability & Updates of Germain Database (e.g. SQL Tables having no updates "recently", etc)
Generate and email report (example attached)
RED: software feature that is broken/failing/unavailable
ORANGE: software feature slow or erroring out
GREEN: available and fast
Example of a Report sent by Germain’s self monitoring script:
Germain status report generated: 2/20/2020 10:00:17 AM
Status | Germain Service | Check | Info |
GermainEngineManager-apsep03050 | LogActivity |
CODE
| |
ActiveMQ | AvailabilityCheck | Status: Running, PID(1296 9708) | |
ActiveMQ | BrokerStats | localhost | Temp Percent: 0 | MemoryPercent: 0 | StorePercent: 0 | |
ActiveMQ | QueueStats | apm.action | QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 28656 | DequeueCount: 28656 | |
ActiveMQ | QueueStats | apm.analytics | QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 15729162 | DequeueCount: 15729162 | |
ActiveMQ | QueueStats | apm.session | QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 0 | DequeueCount: 0 | |
ActiveMQ | QueueStats | apm.storage | QueueSize: 0 | ConsumerCount: 2 | EnqueueCount: 7480961 | DequeueCount: 7480961 | |
ActiveMQ | QueueStats | apm.storage.analytics | QueueSize: 0 | ConsumerCount: 2 | EnqueueCount: 189809 | DequeueCount: 189809 | |
ApacheSolr | AvailabilityCheck | Status: Running, PID(1304 1412 2312) | |
ApacheSolr | EndpointAvailability | Rest Endpoint Response Code: 200 | |
GermainActionServices | AvailabilityCheck | Status: Running, PID(1636) | |
GermainActionServices | LogActivity |
CODE
| |
GermainAggregatorServices | AvailabilityCheck | Status: Running, PID(6720) | |
GermainAggregatorServices | LogActivity |
CODE
| |
GermainAnalyticsServices | AvailabilityCheck | Status: Running, PID(1624) | |
GermainAnalyticsServices | LogActivity |
CODE
| |
GermainConfigServices | EndpointAvailability | Rest Endpoint Response Code: 200 | |
GermainConfigServices | EndpointAvailability | Rest Endpoint Response Code: 200 | |
GermainConfigServices-apsep02522 | LogActivity |
CODE
| |
GermainConfigServices-apsep02523 | LogActivity |
CODE
| |
GermainEnginesProd | AvailabilityCheck | Status: Running, PID(16580 4704) | |
GermainIngestionServices-apsep02522 | LogActivity |
CODE
| |
GermainIngestionServices-apsep02523 | LogActivity |
CODE
| |
GermainQueryServices | EndpointAvailability | Rest Endpoint Response Code: 200 | |
GermainQueryServices | EndpointAvailability | Rest Endpoint Response Code: 200 | |
GermainQueryServices-apsep02522 | LogActivity |
CODE
| |
GermainQueryServices-apsep02523 | LogActivity |
CODE
| |
GermainEngineManager-apsep03069 | LogActivity |
CODE
| |
GermainEngineManagerProd | AvailabilityCheck | Status: Running, PID(121228) | |
GermainSessionTrackingServices | AvailabilityCheck | Status: Running, PID(1060) | |
GermainSessionTrackingServices | LogActivity |
CODE
| |
GermainStorageServices | AvailabilityCheck | Status: Running, PID(1648) | |
GermainStorageServices | LogActivity |
CODE
|
Download
That script is available for download here