Breadcrumbs

Disaster Recovery

warning Disaster Recovery Plan Template

The following steps provide a generic Disaster Recovery plan, intended as a template for planning environment migrations from a Production (PROD) environment to a Recovery/Failover (RECOVERY) environment. As with any template, review and account for specific requirements and special cases before proceeding.

⚙️ 1. Stop All Germain Engines on Production Nodes

Linux example

First, you need to login to each Germain PROD engine servers and run the following commands:

Bash
sudo su
ps -ef | grep manager
kill < PID from step #2 >

After these, wait for all engine processes to stop. It can be verified using the following command:

Bash
ps -ef | grep engine

🔗 2. Update Kafka Url To Point to Recovery Kafka

  • Login to Germain and navigate to following URL: <https://<PROD domain>/germainapm/console/s/#germain.apm.monitoringClient.queueConnectors(Kafka)

  • Update the URL field from your PROD Kafka instance to your RECOVERY instance.

    image-20241008-102417.png
    Kafka connector configuration - Germain UX

🛑 3. Stop All Germain Services on Prod Nodes

Linux example

Login to your Germain PROD server, then run the following commands to kill Germain services:

Bash
sudo su
ps -ef | grep germain | grep <service>  # Example service names: [ aggregation, action, session, analytics, storage ]
kill < PID from step #2 >

For each service, wait and confirm the service is down.

Having the services down, then proceed by shutting down Tomcat:

Bash
cd /germain/apache-tomcat-8/bin   # Navigate to Tomcat's bin folder
./shutdown.sh

You can confirm if Tomcat is down by running the following command:

Bash
ps -ef | grep tomcat

✅ 4. Validate All the Infrastructure Services Are Running As Expected on Recovery

Service

Details

Kafka


Zookeeper


Hazelcast


ElasticSearch


🚀 5. Start All Germain Services on Recovery Nodes

Linux example

The service startup order is the following:

  1. Kafka

  2. Tomcat

  3. Zookeeper

    Bash
    cd /germain/solr/apache-zookeeper-3.7.1
nohup bin/zkServer.sh start # confirm if the sevice is up "ps -ef | grep zookeeper"

  4. ElasticSearch

    cd /germain/elasticsearch-7.17.7
nohup bin/elasticsearch & # confirm if the sevice is up "ps -ef | grep elasticsearch"

  5. Hazelcast

    Bash
    cd /germain/hazelcast-5.3.1
nohup bin/hz start &  # confirm the sevice is up "ps -ef | grep hazelcast"

  6. Storage

  7. Session Tracking

    Bash
    cd /germain/services
nohup bin/sessiontracking-services &
cd /germain/services/var/logs  # verify the status in <service name>.log

  8. Analytics

    Bash
    cd /germain/services
nohup bin/analytics-services &
cd /germain/services/var/logs # verify the status in <service name>.log

  9. Aggregate

    Bash
    cd /germain/services
nohup bin/action-services &
cd /germain/services/var/logs  # verify the status in <service name>.log

  10. Action

    Bash
    cd /germain/services
nohup bin/aggregate-services &
cd /germain/services/var/logs  # verify the status in <service name>.log

🧭 6. Start All Germain Engines on Recovery Nodes

IMPORTANT steps to take BEFORE starting the engines

  • Remove the session.txt file from each engine node before starting.

  • Modify the hostname from the Germain state screen of all nodes before starting. (From PROD to RECOVERY; eg. PROD_***** to RECV_*****)

Linux example

Login to your Germain RECOVERY server, then run the following commands to startup the Germain engines:

sudo su
ps -ef | grep engine  # Confirm no old engines are running
cd /ebay/germain/engine
nohup bin/startEngineManager.sh &

It is possible to check the engine manager status by checking the log file EngineManager.log in the path germain/engine/logs.