Skip to main content
Skip table of contents

Maintenance - Self-monitoring script

Description

 We have completed v2 of the self-monitoring script that monitors and ensures high availability and performance of Germain. That script performs the following:

  • Check Availability of Germain Services (ActiveMQ, ApacheSolr, Action, Aggregation, Analytics, Config, Engines, Ingestion Services, Query Services, Engine Manager, Session Tracking and Storage Services)

  • Check Updates and Errors in Germain’s Logs

  • Check Availability of Germain’s REST Api

  • Check Availability & Updates of Germain Database (e.g. SQL Tables having no updates "recently", etc)

  • Generate and email report (example attached)

    • RED: software feature that is broken/failing/unavailable

    • ORANGE: software feature slow or erroring out

    • GREEN: available and fast

Example of a Report sent by Germain’s self monitoring script:

Germain status report generated: 2/20/2020 10:00:17 AM

Status

Germain Service

Check

Info

GermainEngineManager-apsep03050

LogActivity

CODE
Path:     '\\serverABC\e$\Software\Germain\GermainEngineProd\logs\EngineManager.log'
Modified: 2/20/2020, 9:59:45AM (0.52 minutes ago)
Errors:   7
Warn:     Errors present in recent log.

ActiveMQ

AvailabilityCheck

Status: Running, PID(1296 9708)

ActiveMQ

BrokerStats | localhost

Temp Percent: 0 | MemoryPercent: 0 | StorePercent: 0

ActiveMQ

QueueStats | apm.action

QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 28656 | DequeueCount: 28656

ActiveMQ

QueueStats | apm.analytics

QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 15729162 | DequeueCount: 15729162

ActiveMQ

QueueStats | apm.session

QueueSize: 0 | ConsumerCount: 1 | EnqueueCount: 0 | DequeueCount: 0

ActiveMQ

QueueStats | apm.storage

QueueSize: 0 | ConsumerCount: 2 | EnqueueCount: 7480961 | DequeueCount: 7480961

ActiveMQ

QueueStats | apm.storage.analytics

QueueSize: 0 | ConsumerCount: 2 | EnqueueCount: 189809 | DequeueCount: 189809

ApacheSolr

AvailabilityCheck

Status: Running, PID(1304 1412 2312)

ApacheSolr

EndpointAvailability

Rest Endpoint Response Code: 200

GermainActionServices

AvailabilityCheck

Status: Running, PID(1636)

GermainActionServices

LogActivity

CODE
Path:     'E:\Software\GermainService\var\logs\action-services.log'
Modified: 2/20/2020, 10:00:04AM (0.19 minutes ago)
Errors:   0

GermainAggregatorServices

AvailabilityCheck

Status: Running, PID(6720)

GermainAggregatorServices

LogActivity

CODE
Path:     'E:\Software\GermainService\var\logs\aggregator-services.log'
Modified: 2/20/2020, 10:00:15AM (0.01 minutes ago)
Errors:   0

GermainAnalyticsServices

AvailabilityCheck

Status: Running, PID(1624)

GermainAnalyticsServices

LogActivity

CODE
Path:     '\\serverABC\e$\Software\GermainService\var\logs\analytics-services.log'
Modified: 2/20/2020, 10:00:09AM (0.11 minutes ago)
Errors:   0

GermainConfigServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainConfigServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainConfigServices-apsep02522

LogActivity

CODE
Path:     'E:\Software\apache-tomcat-8\logs\config-services.log'
Modified: 2/20/2020, 10:00:07AM (0.09 minutes ago)
Errors:   0

GermainConfigServices-apsep02523

LogActivity

CODE
Path:     '\\serverABC\e$\Software\apache-tomcat-8\logs\config-services.log'
Modified: 2/20/2020, 9:59:45AM (0.44 minutes ago)
Errors:   0

GermainEnginesProd

AvailabilityCheck

Status: Running, PID(16580 4704)

GermainIngestionServices-apsep02522

LogActivity

CODE
Path:     'E:\Software\apache-tomcat-8\logs\ingestion-services.log'
Modified: 2/20/2020, 10:00:11AM (0.02 minutes ago)
Errors:   0

GermainIngestionServices-apsep02523

LogActivity

CODE
Path:     '\\serverABC\e$\Software\apache-tomcat-8\logs\ingestion-services.log'
Modified: 2/20/2020, 10:00:09AM (0.04 minutes ago)
Errors:   0

GermainQueryServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainQueryServices

EndpointAvailability

Rest Endpoint Response Code: 200

GermainQueryServices-apsep02522

LogActivity

CODE
Path:     'E:\Software\apache-tomcat-8\logs\query-services.log'
Modified: 2/20/2020, 9:59:54AM (0.29 minutes ago)
Errors:   0

GermainQueryServices-apsep02523

LogActivity

CODE
Path:     '\\apsep02523\e$\Software\apache-tomcat-8\logs\query-services.log'
Modified: 2/20/2020, 9:59:52AM (0.32 minutes ago)
Errors:   0

GermainEngineManager-apsep03069

LogActivity

CODE
Path:     '\\apsep03069\e$\Software\Germain\GermainEngineProd\logs\EngineManager.log'
Modified: 2/20/2020, 9:59:58AM (0.32 minutes ago)
Errors:   0

GermainEngineManagerProd

AvailabilityCheck

Status: Running, PID(121228)

GermainSessionTrackingServices

AvailabilityCheck

Status: Running, PID(1060)

GermainSessionTrackingServices

LogActivity

CODE
Path:     '\\serverABC\e$\Software\GermainService\var\logs\session-tracking.log'
Modified: 2/20/2020, 10:00:01AM (0.26 minutes ago)
Errors:   0

GermainStorageServices

AvailabilityCheck

Status: Running, PID(1648)

GermainStorageServices

LogActivity

CODE
Path:     'E:\Software\GermainService\var\logs\storage-services.log'
Modified: 2/20/2020, 10:00:11AM (0.08 minutes ago)
Errors:   0

Download

That script is available for download here

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.