Feature

This document provides instructions on how to operate Germain Application Performance Monitoring 8.x. In addition, it describes methods and best-practices of verifying and trouble-shooting the individual monitoring components.

Germain Enterprise Server

The Germain Enterprise Server application runs as part of a Tomcat application server instance.

Running the Germain Enterprise Server on Windows

On Windows, the Germain Enterprise Server is registered as a system service. To start or stop the service, use the Services Console:

Running the Germain Enterprise Service on Unix

On Unix-based operating systems, the Germain Enterprise Server can be operated using the scripts located in the ./bin directory of the Germain Tomcat installation.

  • To start the server: bin/startup.sh

  • To stop the server: bin/shutdown.sh

Monitoring the State of the Germain Enterprise Server

The Germain Enterprise Server logs status messages to a single location found under the Germain Tomcat root directory: <Tomcat Instance Root>/logs
In this directory, there are several log files that can be used to detect issues related to the Germain Enterprise Server:

  • catalina.out or catalina.yyyy-mm-dd.log

  • germainapmserver-stdout.yyy-mm-dd.log

  • config-services.log

  • query-services.log

  • ingestion-services.log

Germain Engine Manager

Germain Engines that are deployed on a server are controlled by a single Germain Engine Manager. The following section describes how to control the Engine Manager application.

Germain Engine Manager on Windows

On Windows, the Germain Engine Manager is registered as a system service. To start or stop the service, use the Services Console:

 

Germain Engine Manager on Unix

On Unix-based operating systems, the Germain Engine Manager can be operated using the service script provided as part of the engine distribution.

To start the Engine Manager: nohup bin/startEngineManager.sh &
To stop the Engine Manager: 

root@ip-10-0-1-130:/opt/engines# ps -aux | grep apm-engine-manager.jar
root      5505  0.0  0.0  10472   948 pts/0    S+   22:31   0:00 grep --color=auto apm-engine-manager.jar
root      9831  0.0  1.1 4504420 389664 ?      Sl   Nov09   9:38 /usr/bin/java -Dgermain.jvm=/usr/bin/java -Xms256m -Xmx512m -XX:+CrashOnOutOfMemoryError -jar bin/apm-engine-manager.jar
root@ip-10-0-1-130:/opt/engines# kill 9831
root@ip-10-0-1-130:/opt/engines# ps -aux | grep apm-engine-manager.jar
root      5739  0.0  0.0  10472   944 pts/0    S+   22:32   0:00 grep --color=auto apm-engine-manager.jar
root@ip-10-0-1-130:/opt/engines#
CODE

WARNING: Do not use kill -9 . If kill -9 is used, the Engine Manager will not be able to shutdown all the instances of engines for you. (you will need to kill them)

Monitor Germain Engine Manager

The Germain Engine Manager writes status messages and errors to a log found under the Germain Engine directory:
<Engine Root>/logs/EngineManager.log

Manage Germain Engines

The Germain Engines can be controlled through the Germain Workspace on the “APM State” page. To enable or disable a specific engine see the instructions below:

Enabling or Disabling an engine

  • Log into the Workspace as an Administrator

  • Go to APM State

  • Search for the engine by name or browse for the engine in the list

  • Flip the toggle to enabled or disabled

Manage Germain Components

The Germain Engines can be controlled through the Germain Workspace on the “APM State” page.

Enabling or Disabling a Monitor

  • Log into the Workspace as an Administrator

  • Go to APM State

  • Search for the engine by name or browse for the engine in the list

  • select the engine, then click on the enable or disable for the component in question

OR

  • Search for and select the component directly and enable or disable on the top of the details panel

Germain Database

ElasticSearch is the datamart of choice for Germain. PostGres, Oracle, MySQL can also be used by Germain

Monitoring the Alert Log

Germain stores alerts to a central log location that is part of the APM datamart. The following query can be used to review the APM logs:
select * from APM_DATAMART.ALERT_LOG order by TIMESTAMP

Oracle using Materialized Views

The materialized views which are part of the APM datamart schema are configured to be refreshed periodically. Normally, this requires no user interaction; however it is possible for the process to get interrupted. To investigate this situation, the following query can be used:
select JOB, LAST_DATE, FAILURES, BROKEN, WHAT from ALL_JOBS WHERE WHAT LIKE '%dbms_refresh.refresh%'
The result of this query will contain a row for each materialized view refresh job, with a count of failed executions.

ActiveMQ Broker

The ActiveMQ broker runs as a separate process from the Germain Enterprise server. The broker has to be started in advance for the Germain Enterprise and Germain Node servers to communicate with it.

Running ActiveMQ on Windows

On Windows, ActiveMQ can be run as a service using the Program Services Console.

Running ActiveMQ on Unix

On Unix-based operating systems, ActiveMQ can be started from the ActiveMQ directory using the following command:> bin/activemq start xbean:conf/activemq.conf

Message Persistence

Message persistence has a large impact on performance, or throughput of the broker. A non-persistent broker is about 10x faster than one using persistence. We have always deployed Germain as non-persistent to get the maximum performance from each broker instance. For Version: 5.12.3 - 5.13.4 are suspected bad (5.13.3 is known bad) do not use these versions. 5.10.0 and 5.14.3 are known to work.
This is done by setting the persistent attribute of the broker element to "false", e.g.

Network of Brokers

The ActiveMQ network of brokers works a little differently than you might intuitively expect. (At least it worked differently than I expected) Each networkConnector element establishes a one-way connection (this broker becomes a producer for the target broker) that can feed messages. In the example below a network connector named bridge is created to the ActiveMQ broker listening on port 61617 of the gapm-msgbrkr2 server. No messages in the queues named under excludedDestinations will be forwarded. Message in the topic BP.ROUTER.1 will be forwarded when there is at least 1 subscriber for that topic on the destination broker (the meaning of dynamicallyIncludedDestinations). In order to have a networkConnector behave like a regular consumer, you can use staticallyIncludedDestinations (not shown below), which will propagate messages regardless of whether or not the receiving broker has any consumers for the specified queue or topic.


Version Note: 5.12.3 - 5.13.4 are suspected bad (5.13.3 is known bad) do not use these versions. 5.10.0 and 5.14.3 are known to work.