Feature

Germain’s Self-Monitoring (GSM) monitors and ensures high availability and performance of Germain. That script performs the following:

  • Check Availability of Germain Services (ActiveMQ, ApacheSolr, Action, Aggregation, Analytics, Config, Engines, Ingestion Services, Query Services, Engine Manager, Session Tracking and Storage Services)

  • Check Updates and Errors in Germain’s Logs

  • Check Availability of Germain’s REST Api

  • Check Availability & Updates of Germain Database (e.g. SQL Tables having no updates "recently", etc)

  • Generate and email report (example attached)

    • RED: software feature that is broken/failing/unavailable

    • ORANGE: software feature slow or erroring out

    • GREEN: available and fast

Example of a Report sent by GSM script:

Status

Germain Service

Check

Info

GermainEngineManager-apsep03050

LogActivity

Path:     '\\serverABC\e$\Software\Germain\GermainEngineProd\logs\EngineManager.log'
Modified: 2/20/2020, 9:59:45AM (0.52 minutes ago)
Errors:   7
Warn:     Errors present in recent log.
CODE

ActiveMQ

AvailabilityCheck

Status: Running, PID(1296 9708)

ActiveMQ

BrokerStats | localhost

Temp Percent: 0 | MemoryPercent: 0 | StorePercent: 0

ActiveMQ

QueueStats | apm.action

QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 28656 | DequeueCount: 28656

ActiveMQ

QueueStats | apm.analytics

QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 15729162 | DequeueCount: 15729162

ActiveMQ

QueueStats | apm.session

QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 0 | DequeueCount: 0

ActiveMQ

QueueStats | apm.storage

QueueSize: 0 | ConsumerCount: 2 | EnqueueCount: 7480961 | DequeueCount: 7480961

ActiveMQ

QueueStats | apm.storage.analytics

QueueSize: 0 | ConsumerCount: 2 | EnqueueCount: 189809 | DequeueCount: 189809

ApacheSolr

AvailabilityCheck

Status: Running, PID(1304 1412 2312)

ApacheSolr

EndpointAvailability

Rest Endpoint Response Code: 200

GermainActionServices

AvailabilityCheck

Status: Running, PID(1636)

GermainActionServices

LogActivity

Path:     'E:\Software\GermainService\var\logs\action-services.log'
Modified: 2/20/2020, 10:00:04AM (0.19 minutes ago)
Errors:   0
CODE

GermainAggregatorServices

AvailabilityCheck

Status: Running, PID(6720)

GermainAggregatorServices

LogActivity

Path:     'E:\Software\GermainAPMService\var\logs\aggregator-services.log'
Modified: 2/20/2020, 10:00:15AM (0.01 minutes ago)
Errors:   0
CODE

GermainAnalyticsServices

AvailabilityCheck

Status: Running, PID(1624)

GermainAnalyticsServices

LogActivity

Path:     '\\serverABC\e$\Software\GermainAPMService\var\logs\analytics-services.log'
Modified: 2/20/2020, 10:00:09AM (0.11 minutes ago)
Errors:   0
CODE

GermainAPMConfigServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainAPMConfigServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainAPMConfigServices-apsep02522

LogActivity

Path:     'E:\Software\apache-tomcat-8\logs\config-services.log'
Modified: 2/20/2020, 10:00:07AM (0.09 minutes ago)
Errors:   0
CODE

GermainAPMConfigServices-apsep02523

LogActivity

Path:     '\\serverABC\e$\Software\apache-tomcat-8\logs\config-services.log'
Modified: 2/20/2020, 9:59:45AM (0.44 minutes ago)
Errors:   0
CODE

GermainAPMEnginesProd

AvailabilityCheck

Status: Running, PID(16580 4704)

GermainAPMIngestionServices-apsep02522

LogActivity

Path:     'E:\Software\apache-tomcat-8\logs\ingestion-services.log'
Modified: 2/20/2020, 10:00:11AM (0.02 minutes ago)
Errors:   0
CODE

GermainAPMIngestionServices-apsep02523

LogActivity

Path:     '\\serverABC\e$\Software\apache-tomcat-8\logs\ingestion-services.log'
Modified: 2/20/2020, 10:00:09AM (0.04 minutes ago)
Errors:   0
CODE

GermainAPMQueryServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainAPMQueryServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainAPMQueryServices-apsep02522

LogActivity

Path:     'E:\Software\apache-tomcat-8\logs\query-services.log'
Modified: 2/20/2020, 9:59:54AM (0.29 minutes ago)
Errors:   0
CODE

GermainAPMQueryServices-apsep02523

LogActivity

Path:     '\\apsep02523\e$\Software\apache-tomcat-8\logs\query-services.log'
Modified: 2/20/2020, 9:59:52AM (0.32 minutes ago)
Errors:   0
CODE

GermainEngineManager-apsep03069

LogActivity

Path:     '\\apsep03069\e$\Software\Germain\GermainAPMEngineProd\logs\EngineManager.log'
Modified: 2/20/2020, 9:59:58AM (0.32 minutes ago)
Errors:   0
CODE

GermainEngineManagerProd

AvailabilityCheck

Status: Running, PID(121228)

GermainSessionTrackingServices

AvailabilityCheck

Status: Running, PID(1060)

GermainSessionTrackingServices

LogActivity

Path:     '\\serverABC\e$\Software\GermainAPMService\var\logs\session-tracking.log'
Modified: 2/20/2020, 10:00:01AM (0.26 minutes ago)
Errors:   0
CODE

GermainStorageServices

AvailabilityCheck

Status: Running, PID(1648)

GermainStorageServices

LogActivity

Path:     'E:\Software\GermainAPMService\var\logs\storage-services.log'
Modified: 2/20/2020, 10:00:11AM (0.08 minutes ago)
Errors:   0
CODE

Download GSM

That script is available for download here