Maintenance - Self-monitoring script

Description

We have completed v2 of the self-monitoring script that monitors and ensures high availability and performance of Germain. That script performs the following:

Check Availability of Germain Services (ActiveMQ, ApacheSolr, Action, Aggregation, Analytics, Config, Engines, Ingestion Services, Query Services, Engine Manager, Session Tracking and Storage Services)
Check Updates and Errors in Germain’s Logs
Check Availability of Germain’s REST Api
Check Availability & Updates of Germain Database (e.g. SQL Tables having no updates "recently", etc)
Generate and email report (example attached)
- RED: software feature that is broken/failing/unavailable
- ORANGE: software feature slow or erroring out
- GREEN: available and fast

Example of a Report sent by Germain’s self monitoring script:

Germain status report generated: 2/20/2020 10:00:17 AM

Status	Germain Service	Check	Info
	GermainEngineManager-apsep03050	LogActivity	CODE `Path: '\\serverABC\e$\Software\Germain\GermainEngineProd\logs\EngineManager.log' Modified: 2/20/2020, 9:59:45AM (0.52 minutes ago) Errors: 7 Warn: Errors present in recent log.`
	ActiveMQ	AvailabilityCheck	Status: Running, PID(1296 9708)
	ActiveMQ	BrokerStats \| localhost	Temp Percent: 0 \| MemoryPercent: 0 \| StorePercent: 0
	ActiveMQ	QueueStats \| apm.action	QueueSize: 0 \| ConsumerCount: 1 \| EnqueueCount: 28656 \| DequeueCount: 28656
	ActiveMQ	QueueStats \| apm.analytics	QueueSize: 0 \| ConsumerCount: 1 \| EnqueueCount: 15729162 \| DequeueCount: 15729162
	ActiveMQ	QueueStats \| apm.session	QueueSize: 0 \| ConsumerCount: 1 \| EnqueueCount: 0 \| DequeueCount: 0
	ActiveMQ	QueueStats \| apm.storage	QueueSize: 0 \| ConsumerCount: 2 \| EnqueueCount: 7480961 \| DequeueCount: 7480961
	ActiveMQ	QueueStats \| apm.storage.analytics	QueueSize: 0 \| ConsumerCount: 2 \| EnqueueCount: 189809 \| DequeueCount: 189809
	ApacheSolr	AvailabilityCheck	Status: Running, PID(1304 1412 2312)
	ApacheSolr	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainActionServices	AvailabilityCheck	Status: Running, PID(1636)
	GermainActionServices	LogActivity	CODE `Path: 'E:\Software\GermainService\var\logs\action-services.log' Modified: 2/20/2020, 10:00:04AM (0.19 minutes ago) Errors: 0`
	GermainAggregatorServices	AvailabilityCheck	Status: Running, PID(6720)
	GermainAggregatorServices	LogActivity	CODE `Path: 'E:\Software\GermainService\var\logs\aggregator-services.log' Modified: 2/20/2020, 10:00:15AM (0.01 minutes ago) Errors: 0`
	GermainAnalyticsServices	AvailabilityCheck	Status: Running, PID(1624)
	GermainAnalyticsServices	LogActivity	CODE `Path: '\\serverABC\e$\Software\GermainService\var\logs\analytics-services.log' Modified: 2/20/2020, 10:00:09AM (0.11 minutes ago) Errors: 0`
	GermainConfigServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainConfigServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainConfigServices-apsep02522	LogActivity	CODE `Path: 'E:\Software\apache-tomcat-8\logs\config-services.log' Modified: 2/20/2020, 10:00:07AM (0.09 minutes ago) Errors: 0`
	GermainConfigServices-apsep02523	LogActivity	CODE `Path: '\\serverABC\e$\Software\apache-tomcat-8\logs\config-services.log' Modified: 2/20/2020, 9:59:45AM (0.44 minutes ago) Errors: 0`
	GermainEnginesProd	AvailabilityCheck	Status: Running, PID(16580 4704)
	GermainIngestionServices-apsep02522	LogActivity	CODE `Path: 'E:\Software\apache-tomcat-8\logs\ingestion-services.log' Modified: 2/20/2020, 10:00:11AM (0.02 minutes ago) Errors: 0`
	GermainIngestionServices-apsep02523	LogActivity	CODE `Path: '\\serverABC\e$\Software\apache-tomcat-8\logs\ingestion-services.log' Modified: 2/20/2020, 10:00:09AM (0.04 minutes ago) Errors: 0`
	GermainQueryServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainQueryServices	EndpointAvailability	Rest Endpoint Response Code: 200
	GermainQueryServices-apsep02522	LogActivity	CODE `Path: 'E:\Software\apache-tomcat-8\logs\query-services.log' Modified: 2/20/2020, 9:59:54AM (0.29 minutes ago) Errors: 0`
	GermainQueryServices-apsep02523	LogActivity	CODE `Path: '\\apsep02523\e$\Software\apache-tomcat-8\logs\query-services.log' Modified: 2/20/2020, 9:59:52AM (0.32 minutes ago) Errors: 0`
	GermainEngineManager-apsep03069	LogActivity	CODE `Path: '\\apsep03069\e$\Software\Germain\GermainEngineProd\logs\EngineManager.log' Modified: 2/20/2020, 9:59:58AM (0.32 minutes ago) Errors: 0`
	GermainEngineManagerProd	AvailabilityCheck	Status: Running, PID(121228)
	GermainSessionTrackingServices	AvailabilityCheck	Status: Running, PID(1060)
	GermainSessionTrackingServices	LogActivity	CODE `Path: '\\serverABC\e$\Software\GermainService\var\logs\session-tracking.log' Modified: 2/20/2020, 10:00:01AM (0.26 minutes ago) Errors: 0`
	GermainStorageServices	AvailabilityCheck	Status: Running, PID(1648)
	GermainStorageServices	LogActivity	CODE `Path: 'E:\Software\GermainService\var\logs\storage-services.log' Modified: 2/20/2020, 10:00:11AM (0.08 minutes ago) Errors: 0`

Download

That script is available for download here