Disaster Recovery

Disaster Recovery Plan Template

This runbook provides a generic framework for failing over GermainUX from a Production environment (PROD) to a Recovery environment (RECOVERY).

It must be adapted and tested for each deployment. Service names, paths, versions, dependencies, authentication methods, and startup procedures vary by environment.

📋 Pre-Failover Checklist

Record the following information before initiating recovery:

Confirm that the RECOVERY environment contains compatible GermainUX configurations and software versions.

🛑 1. Stop GermainUX Engines in PROD

Stop all GermainUX Engines gracefully to prevent them from continuing to send data or execute automation during the cutover.

For each PROD Engine server:

• Identify the Engine and Engine Manager processes.

• Use the approved service manager or GermainUX shutdown procedure.

• Confirm that all processes have stopped.

• Record the shutdown result in the incident or change record.

Example verification commands:

pgrep -af "engine|manager"

If the processes are managed by systemd, use the applicable service unit:

sudo systemctl stop <germain-engine-unit>
sudo systemctl status <germain-engine-unit>

Do not use an unconditional process kill as the standard shutdown method. If graceful shutdown fails, follow the approved escalation procedure and preserve relevant logs before terminating the process.

⚙️ 2. Stop Germain Enterprise Services in PROD

Stop the Germain Enterprise services using the approved service-management mechanism.

Services may include:

For services managed by systemd:

sudo systemctl stop <germain-service-unit>
sudo systemctl status <germain-service-unit>

For application-container deployments, use the shutdown script associated with the installed version:

cd <TOMCAT_HOME>/bin
./shutdown.sh

Verify that no applicable GermainUX or application-container processes remain:

pgrep -af "germain|tomcat"

Use the shutdown order documented for your environment. Do not terminate shared infrastructure until dependent services have stopped cleanly.

✅ 3. Confirm Data and Infrastructure Readiness

Before starting GermainUX in RECOVERY, validate the readiness of every required dependency.

If Kafka uses ZooKeeper, start and validate ZooKeeper before starting Kafka. Kafka deployments using KRaft do not require ZooKeeper. Follow the dependency order applicable to the installed architecture.

🔧 4. Update the RECOVERY Kafka Connector

Update the GermainUX configuration used by the RECOVERY environment so it references the RECOVERY Kafka service.

Open:

https://<GERMAIN_DOMAIN>/germainapm/console/s/#germain.apm.monitoringClient.queueConnectors(Kafka)

Then:

1. Select the Kafka connector used by RECOVERY.

2. Replace the PROD broker URL with the RECOVERY broker URL.

3. Verify authentication, TLS, certificates, ports, and topic settings.

4. Save the configuration.

5. Test connectivity before starting data-producing services.

Screenshot: Kafka connector configuration in GermainUX

Avoid modifying the active PROD connector before PROD processing has been stopped unless the approved failover design specifically requires it.

🚀 5. Start Infrastructure in RECOVERY

Start and validate infrastructure dependencies in the order required by your architecture.

A typical dependency sequence is:

1. ZooKeeper, if required

2. Kafka

3. Database or Elasticsearch

4. Hazelcast

5. Application container

Do not rely on hard-coded installation paths or version numbers. Use the paths and service units documented for the RECOVERY environment.

For each dependency:

1. Start the service.

2. Confirm that its process is running.

3. Review its health endpoint or cluster status.

4. Check its logs for startup errors.

5. Confirm connectivity from the GermainUX RECOVERY nodes.

6. Continue only after the dependency is healthy.

Example:

sudo systemctl start <infrastructure-unit>
sudo systemctl status <infrastructure-unit>

⏯️ 6. Start Germain Enterprise Services in RECOVERY

Start Germain Enterprise services according to the dependency order validated for your installed version.

A typical sequence is:

1. Storage Service

2. Session Tracking Service

3. Analytics Service

4. Aggregation Service

5. Action Service

6. Ingestion Service

7. REST or other application services

Example:

sudo systemctl start <germain-service-unit>
sudo systemctl status <germain-service-unit>

If the environment uses GermainUX startup scripts instead of systemd, execute the scripts defined in the RECOVERY operations documentation.

After starting each service:

• Confirm that the process remains running.

• Review its logs.

• Verify connectivity to Kafka, Hazelcast, and the datastore.

• Confirm that no repeated retries or authentication failures occur.

• Confirm that the service appears healthy in GermainUX Workspace > Germain > State.

Do not proceed if a required service is unhealthy.

⚙️ 7. Prepare RECOVERY Engines

Before starting the RECOVERY Engines:

1. Confirm that each Engine points to the RECOVERY environment.

2. Verify its credentials, certificates, ports, and queue configuration.

3. Confirm that its node identity is unique.

4. Update any environment-specific hostname or node references from PROD to RECOVERY.

5. Review component schedules to prevent duplicate executions.

6. Confirm that PROD Engines remain stopped.

📁 Session-State File

If your GermainUX version and deployment procedure require resetting session.txt:

1. Confirm the exact file path.

2. Stop the Engine.

3. Create a recoverable backup of the file.

4. Remove or rename only the validated file.

5. Record the operation.

Example:

mv <ENGINE_PATH>/session.txt <ENGINE_PATH>/session.txt.pre-failover

Do not use a broad deletion command or an unresolved path.

🆔 Update Node Identity

In GermainUX Workspace > Germain > State, update the applicable node or hostname references from the PROD naming convention to the RECOVERY naming convention.

Example:

PROD_<NODE_NAME> → RECOVERY_<NODE_NAME>

Verify that the change does not create duplicate node identities.

▶️ 8. Start RECOVERY Engines

Start each Engine using the approved service manager or startup procedure.

Example:

sudo systemctl start <germain-engine-unit>
sudo systemctl status <germain-engine-unit>

For each Engine:

1. Confirm that the process remains running.

2. Verify registration in the GermainUX State dashboard.

3. Confirm that deployed components start successfully.

4. Review Engine and component logs.

5. Confirm that data is reaching the RECOVERY Kafka and GermainUX services.

6. Verify that scheduled actions are not duplicated.

🔍 9. Validate the RECOVERY Environment

Perform technical and functional validation before declaring recovery complete.

🖥️ Platform Validation

• All required Germain Enterprise services are healthy.

• Nodes, Engines, agents, and components are connected.

• Kafka queues are processing without a continuously growing backlog.

• Hazelcast members form the expected cluster.

• The datastore is healthy and writable.

• REST endpoints respond successfully.

• No critical startup or connectivity errors appear in logs.

📊 Data Validation

• New monitoring data is ingested.

• Aggregated data is generated.

• User sessions are tracked.

• Dashboards display current data.

• Historical data required by the recovery plan is available.

• Data timestamps, environments, and hostnames are correct.

🤖 Analytics and Automation Validation

• SLAs are evaluated.

• Alerts and reports can be generated.

• Scheduled actions execute only once.

• Synthetic scenarios run successfully.

• Automation targets the RECOVERY environment.

• No action can unintentionally affect PROD.

👥 User Validation

• Authorized users can log in.

• Roles, teams, and permissions operate correctly.

• Dashboards and Session Replay are accessible as expected.

• A representative monitoring workflow completes successfully.

🏳️ 10. Complete the Cutover

After validation:

1. Update DNS, load balancers, integrations, or monitoring endpoints according to the approved cutover plan.

2. Notify stakeholders that RECOVERY is active.

3. Record the recovery time and any observed data loss.

4. Preserve relevant logs and evidence.

5. Continue heightened monitoring for the defined stabilization period.

6. Keep PROD services stopped until the recovery owner authorizes another change.

⏪ Rollback Plan

Define rollback criteria before beginning the failover.

A rollback may be required if:

• Critical GermainUX services cannot start.

• Data replication is incomplete or inconsistent.

• Automation could target the wrong environment.

• Required monitoring data cannot be ingested or stored.

• Security, authentication, or access controls fail validation.

• Recovery cannot be completed within the approved RTO.

The rollback procedure must specify how to stop RECOVERY safely, restore the previous connector and endpoint configuration, validate PROD, and prevent duplicate data processing or automation.

📝 Post-Recovery Review

After stabilization:

• Document the incident timeline.

• Record actual RTO and RPO performance.

• Identify failed or manual steps.

• Update service names, dependencies, paths, and owners.

• Review monitoring and alert coverage.

• Confirm that backups and replication have resumed.

• Schedule the next disaster-recovery test.

• Obtain technical and business sign-off.

This template does not replace environment-specific operational procedures. Test the completed runbook regularly in a non-production or controlled recovery exercise.

ℹ️ Get Help

The Germain Team can help you set this up. Contact GermainUX Support.