Sentinel (Germain UX's self-monitoring)

Feature

Sentinel is a monitoring system, spun off Germain UX, ensures high availability and performance of Germain UX. The Sentinel script performs the following tasks:

Monitoring features:
- Connect to Germain infrastructure services (Zookeeper, ActiveMQ) and record availability / outages as well as metrics (ex: queue usage).
- Monitor configurable list of OS services, report on availability, CPU + memory usage.
- Monitor configurable list of log files, check last write time, check for errors or warnings based on configurable threshold.
- Monitor configurable HTTP endpoints, report status + response.
Summarize all findings in a single email report, some configuration around under which conditions to send the report email

The report categorizes the status of each software feature as follows:

RED: Indicates a software feature that is broken, failing, or unavailable.
ORANGE: Indicates a software feature that is slow or experiencing errors.
GREEN: Indicates a software feature that is available and fast.

The attached example provides a sample report sent by the Sentinel script.

Please note that the specific details and format of the report may vary based on the implementation and configuration of the Sentinel system.

Example of a Report sent by Sentinel script:

Status	Germain Service	Check	Info
	GermainEngineManager-apsep03050	LogActivity	CODE `Path: '\\serverABC\e$\Software\Germain\GermainEngineProd\logs\EngineManager.log' Modified: 2/20/2020, 9:59:45AM (0.52 minutes ago) Errors: 7 Warn: Errors present in recent log.`
	ActiveMQ	AvailabilityCheck	Status: Running, PID(1296 9708)
	ActiveMQ	BrokerStats \| localhost	Temp Percent: 0 \| MemoryPercent: 0 \| StorePercent: 0
	ActiveMQ	QueueStats \| apm.action	QueueSize: 0 \| ConsumerCount: 1 \| EnqueueCount: 28656 \| DequeueCount: 28656
	ActiveMQ	QueueStats \| apm.analytics	QueueSize: 0 \| ConsumerCount: 1 \| EnqueueCount: 15729162 \| DequeueCount: 15729162
	ActiveMQ	QueueStats \| apm.session	QueueSize: 0 \| ConsumerCount: 1 \| EnqueueCount: 0 \| DequeueCount: 0
	ActiveMQ	QueueStats \| apm.storage	QueueSize: 0 \| ConsumerCount: 2 \| EnqueueCount: 7480961 \| DequeueCount: 7480961
	ActiveMQ	QueueStats \| apm.storage.analytics	QueueSize: 0 \| ConsumerCount: 2 \| EnqueueCount: 189809 \| DequeueCount: 189809
	GermainActionServices	AvailabilityCheck	Status: Running, PID(1636)
	GermainActionServices	LogActivity	CODE `Path: 'E:\Software\GermainService\var\logs\action-services.log' Modified: 2/20/2020, 10:00:04AM (0.19 minutes ago) Errors: 0`
	GermainAggregatorServices	AvailabilityCheck	Status: Running, PID(6720)
	GermainAggregatorServices	LogActivity	CODE `Path: 'E:\Software\GermainAPMService\var\logs\aggregator-services.log' Modified: 2/20/2020, 10:00:15AM (0.01 minutes ago) Errors: 0`
	GermainAnalyticsServices	AvailabilityCheck	Status: Running, PID(1624)
	GermainAnalyticsServices	LogActivity	CODE `Path: '\\serverABC\e$\Software\GermainAPMService\var\logs\analytics-services.log' Modified: 2/20/2020, 10:00:09AM (0.11 minutes ago) Errors: 0`
	GermainAPMConfigServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainAPMConfigServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainAPMConfigServices-apsep02522	LogActivity	CODE `Path: 'E:\Software\apache-tomcat-8\logs\config-services.log' Modified: 2/20/2020, 10:00:07AM (0.09 minutes ago) Errors: 0`
	GermainAPMConfigServices-apsep02523	LogActivity	CODE `Path: '\\serverABC\e$\Software\apache-tomcat-8\logs\config-services.log' Modified: 2/20/2020, 9:59:45AM (0.44 minutes ago) Errors: 0`
	GermainAPMEnginesProd	AvailabilityCheck	Status: Running, PID(16580 4704)
	GermainAPMIngestionServices-apsep02522	LogActivity	CODE `Path: 'E:\Software\apache-tomcat-8\logs\ingestion-services.log' Modified: 2/20/2020, 10:00:11AM (0.02 minutes ago) Errors: 0`
	GermainAPMIngestionServices-apsep02523	LogActivity	CODE `Path: '\\serverABC\e$\Software\apache-tomcat-8\logs\ingestion-services.log' Modified: 2/20/2020, 10:00:09AM (0.04 minutes ago) Errors: 0`
	GermainAPMQueryServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainAPMQueryServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainAPMQueryServices-apsep02522	LogActivity	CODE `Path: 'E:\Software\apache-tomcat-8\logs\query-services.log' Modified: 2/20/2020, 9:59:54AM (0.29 minutes ago) Errors: 0`
	GermainAPMQueryServices-apsep02523	LogActivity	CODE `Path: '\\apsep02523\e$\Software\apache-tomcat-8\logs\query-services.log' Modified: 2/20/2020, 9:59:52AM (0.32 minutes ago) Errors: 0`
	GermainEngineManager-apsep03069	LogActivity	CODE `Path: '\\apsep03069\e$\Software\Germain\GermainAPMEngineProd\logs\EngineManager.log' Modified: 2/20/2020, 9:59:58AM (0.32 minutes ago) Errors: 0`
	GermainEngineManagerProd	AvailabilityCheck	Status: Running, PID(121228)
	GermainSessionTrackingServices	AvailabilityCheck	Status: Running, PID(1060)
	GermainSessionTrackingServices	LogActivity	CODE `Path: '\\serverABC\e$\Software\GermainAPMService\var\logs\session-tracking.log' Modified: 2/20/2020, 10:00:01AM (0.26 minutes ago) Errors: 0`
	GermainStorageServices	AvailabilityCheck	Status: Running, PID(1648)
	GermainStorageServices	LogActivity	CODE `Path: 'E:\Software\GermainAPMService\var\logs\storage-services.log' Modified: 2/20/2020, 10:00:11AM (0.08 minutes ago) Errors: 0`

Service: Enterprise

Feature Availability: 2023.1