> ## Documentation Index
> Fetch the complete documentation index at: https://docs.germainux.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Find the Release Behind a Production Problem Across MuleSoft, NGINX and PostgreSQL

> Find out whether a release caused the errors, slowdown or outage you see, on which layer, and whether to roll back, from the API to the database.

Components
[`Engine`](https://docs.germainux.com/deploy/germain-ux-engine)

> [card] [What GermainUX Watches](#what-germainux-watches)
>
> Errors, speed, resources and releases

> [card] [Find the Cause](#find-the-cause)
>
> Three worked examples

> [card] [Alert and Act](#alert-and-act)
>
> Catch the next one sooner

> [card] [Set It Up](#set-it-up)
>
> Layers, markers and tests

## Overview

Something went wrong in production, and a release went out this morning. The first question in the incident call is always the same: did the release cause it, and should we roll back? Developers, operations and database teams each check their own layer, and the release that caused it may not be the one everybody suspects.

A health check answers only part of it. A bad release can fail half the calls of one API, slow down two pages without a single error, or leak resources for hours before anything breaks. Each one shows on a different layer and at a different time.

GermainUX puts the releases on the same timeline as the KPIs of every layer. It compares the period after the release with the period before, shows which API, endpoint or host changed, and lets you rule out what stayed normal. You decide to roll back or fix forward from evidence, and then confirm that it worked.

Alert
Aggregate Analysis
Release Marker
Affected Layer
Roll Back or Fix
Confirm

This page uses a MuleSoft integration layer, NGINX web servers and a PostgreSQL database as the example. The same approach works for any stack GermainUX monitors, and for releases that affect users directly, such as the [theme release that slowed a Shopify product page](https://docs.germainux.com/use-cases/conversion-drop-analysis.md#example-a-theme-release-slows-the-product-page).

### Questions It Answers

* Did this morning's release cause the problem?
* Should we roll back, or fix forward?
* Which layer changed: the API, the web servers or the database?
* How many orders or users did the release affect?
* Is a release slowing things down without raising any error?
* Did the rollback bring everything back to normal?

## What GermainUX Watches

GermainUX keeps each signal as a KPI, so the analysis can compare the time after a release with the time before.

| Signal | What it tells you | Where it comes from |
| - | - | - |
| **API transactions and errors** | Failure rate, error type and response time of each API and operation | [MuleSoft Observability](https://docs.germainux.com/integrations/mulesoft-monitoring.md) |
| **Web requests** | Response time and status of each route, by server | [NGINX monitoring](https://docs.germainux.com/integrations/nginx-monitoring.md) |
| **Database resources** | Connections, availability, locks and I/O | [PostgreSQL monitoring](https://docs.germainux.com/integrations/postgresql-monitoring.md) |
| **New errors** | Error types that never appeared before, a common sign of a regression | [New error detection](https://docs.germainux.com/analytics/ai-driven-insights.md#new-error-detection-and-analysis) |
| **Releases and changes** | Deployments, rollouts and configuration reloads, shown as markers on the timeline | [KPI relationships shown as markers](https://docs.germainux.com/analytics/kpis/kpi-relationships.md#marker) |
| **Real users** | The pages, sessions and business processes the release affected | [Session Replay](https://docs.germainux.com/analytics/user-monitoring-and-replay.md), [business processes](https://docs.germainux.com/analytics/business-process-workflow-monitoring.md) |

## Find the Cause

The investigation follows the same path for every release:

1. Open [Aggregate Analysis](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md#aggregate-view-vs-instance-view) on the KPI that raised the alert, from before the release to now. GermainUX compares the period with its baseline.
2. Find the release on the [Relationships](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md#relationships) chart. A marker, or the first change in a KPI, shows when it started.
3. Read the [Leading Factors](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md#leading-factors) to see which API, operation, route or host changed, and check the [Related KPIs](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md#related-kpis) on the other layers.
4. Rule out what stayed normal. It tells you whether the release is the only suspect.
5. Roll back or fix, then run the same analysis on the period after.

The three examples below come from simulated data, and their numbers belong to the example. Each release breaks a different layer, in a different way.

### Example: A Payments API Release Fails Almost Half the Payments

At 08:00, a new version of the payments API rolls out on MuleSoft. Five minutes later, payments start to fail. In this example, 45% of payment calls fail, against less than 1% before.

**1. Spot the failures**

Open the analysis of the MuleSoft API transactions for the payments API. Failures of `POST /api/payments` jump at 08:05, with a new error type, `MULE:EXPRESSION`. Response time doesn't change, so an alert on response time alone wouldn't fire.

**2. Find the release**

The release marker of the payments API sits at 08:00. Its availability shows one of its two replicas starting at 08:00 and 08:05: the rollout of the new version. The failures begin as the new replicas take traffic. Hover over the marker to see what the release changed. In this example, version 1.4.2 of the payments API changed the payment flow and its card mapping.

![Payment Failures analysis of payments-papi where failed calls jump right after the release marker at 08:00 and fall back after the rollback marker at 09:15, with POST /api/payments ranked first in Leading Factors](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/payments-release-failures.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=beeb622eabd93f7d0e7a4a208440ecba)

Payment Failures After the Payments API Release - GermainUX

![Release marker at 08:00 on the Payment Failures chart, its tooltip showing Deployment Run DEP-9412, payments-papi 1.4.2, completed on cloudhub-prod, with the changed files post-payment.xml, card-mapping.dwl and pom.xml](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/payments-release-marker.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=4bc9e3e1b230b4f824763fcc666bd6fa)

Payments API Release Details on the Release Marker - GermainUX

**3. Measure the impact**

Add the APIs that call it. The orders API and the EDI gateway now fail with `502` on the calls that need a payment, so about 45% of order placements fail too.

![Analysis of MuleSoft calls that returned 502 after the payments API release, with Leading Factors ranking orders-api and POST /api/orders first, followed by edi-gateway](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/payments-release-caller-failures.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=b498d9e710be0a5212aa1ef46eec8992)

Calls Failing With 502 After the Payments API Release - GermainUX

**4. Rule out the rest**

The customer and inventory APIs, reading a payment, and the runtime's JVM metrics all stay at their baseline. Only the new version's payment call changed.

Roll back. The replicas show the rollback starting at 09:15, and failures are back below 1% by 09:20. The error type tells the developers where to look before the next attempt.

### Example: A Storefront Release Makes Search Four Times Slower

At 11:00, a storefront release changes the search and product-review pages. Nothing fails: every request still returns `200`. But in this example, search takes about 1 second instead of 0.3, and reviews 0.7 seconds instead of 0.18.

**1. Spot the slowdown**

Open the analysis of the inbound HTTP requests of the storefront for the day. Response time rises from 11:00 and stays high, while errors and traffic stay flat.

**2. Find the release**

The storefront's release marker sits two minutes before the slowdown, and both storefront servers log a configuration reload at the same time. Hover over the marker to see what the release changed. In this example, it changed the search and review scripts and the server configuration.

![Search Response Time analysis of the storefront where the average response time of /search rises at the release marker at 11:00 and stays high for the afternoon](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/storefront-release-slowdown.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=3e468941af2657fe902c237da9a49c43)

Search Response Time After the Storefront Release - GermainUX

![Release marker at 10:58 on the Search Response Time chart, its tooltip showing Web Release DEP-9377, storefront search and reviews, completed on web-01, with the changed files storefront.conf, results.js, facets.js and widget.js](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/storefront-release-marker.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=e3119983b4ea3dbffc11ffe6828a886b)

Storefront Release Details on the Release Marker - GermainUX

**3. Read what changed**

Filter the Segmentation Breakdown to the HTTP paths. In this example, `/search` averages 1.1 seconds and stands out as an outlier, while the other storefront routes stay below 0.2 seconds. The product-review API slowed down too.

![Storefront Response Time analysis with the Segmentation Breakdown filtered to HTTP paths, where the /search segment is outlined in red, averaging 1.141 seconds and flagged as an outlier, while the other routes stay below 0.16 seconds](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/storefront-slow-routes.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=50bf28fbc537dd514d2fd46ccfe01827)

Search Stands Out Among the Storefront Routes - GermainUX

**4. Rule out the rest**

The customer portal on the third server, the other storefront routes and the static files stay at their baseline. The servers aren't overloaded; two routes changed.

Revert or fix the search and review changes, then compare the response time of the two routes with the period before the release.

### Example: A Release Leaks Database Connections for Three Hours

At 06:45, an application release goes out. Nothing fails for more than two hours. Then, at about 09:20, the database starts refusing new clients.

**1. Spot the outage**

Database availability on the primary drops to 0, with `FATAL: sorry, too many clients already`. While the database refuses clients, GermainUX gets no connection counts, so the database connections KPI has a gap for that period.

![Database Availability drillthrough of pg-prod-01 where every reading from 09:20 to 09:55 is 0 and readings return to 100 from 10:00](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/orders-database-outage.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=32b241db08f172b1c4aac08d5f3c16d1)

Primary Database Availability Drops to 0 - GermainUX

**2. Look before the outage**

Widen the analysis of database connections to the whole morning. In this example, they normally follow traffic, from about 120 in the morning to 260 at 10:00. From 06:45, they climb steadily, by 5 to 6 a minute, whatever the traffic. They pass 1,000 at about 09:05, and the connections SLA is breached from 09:10, 10 minutes before the outage.

**3. Find the release**

The climb starts at 06:45, exactly at the marker of the order service's release, more than two hours before anything failed. Without the marker, you'd look for the cause around the outage, not at the start of the climb. Hover over the marker to see what the release changed. In this example, version 3.12.0 of the order service changed its order repository and its application configuration.

![Database Connections analysis of pg-prod-01 where connections climb from the release marker at 06:45 to about 1,000 before the outage](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/orders-release-connection-leak.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=f335bfc00e094d6706a60a794ae5d9ab)

Database Connections Climbing From the Order Service Release - GermainUX

![Release marker at 06:45 on the Database Connections chart, its tooltip showing Deployment Run DEP-9341, orders-service 3.12.0, completed on app-prod-01, with the changed files OrderRepository.java and application.yml](https://mintcdn.com/germain-ux/K5T43Jp0EGmnztW6/images/main/release-impact-analysis/orders-release-marker.png?fit=max\&auto=format\&n=K5T43Jp0EGmnztW6\&q=85\&s=d89f3c7527cbe92598e06a6af673f1ad)

Order Service Release Details on the Release Marker - GermainUX

**4. Rule out the rest**

The standby, deadlocks, dead rows and I/O stay at their baseline. It isn't load or a slow query; connections are opened and never returned.

Restarting the connection pool brings the database back at 10:00, but only buys time. Roll back or fix the release, then check that connections follow traffic again.

## Alert and Act

Each example points to a signal you can catch sooner next time:

* **[Percent change SLAs](https://docs.germainux.com/analytics/slas.md#percent-change-sla)** on error rates and response times, so a regression after a release raises an alert even when no fixed threshold is crossed
* **[New error detection](https://docs.germainux.com/analytics/ai-driven-insights.md#new-error-detection-and-analysis)**, so an error type that never appeared before is flagged as soon as it does
* **[SLAs](https://docs.germainux.com/analytics/slas.md)** and [outlier detection](https://docs.germainux.com/analytics/outlier-detection.md) on resources that leak slowly, such as connections and memory
* **[Synthetic user scenarios](https://docs.germainux.com/automation/user-click-synthetic-user-scenario.md)** and [HTTP checks](https://docs.germainux.com/automation/synthetic-http-execution.md) that run the critical paths right after each deployment
* **[Automated actions](https://docs.germainux.com/automation/automated-actions.md#automated-technology-remediation-and-recovery)** that roll back a deployment, with approval before they run
* **[Alerts](https://docs.germainux.com/automation/notifications/create-an-alert.md)** to the team that shipped the release, by email, [Microsoft Teams](https://docs.germainux.com/automation/notifications/microsoft-teams-notification.md) or [Slack](https://docs.germainux.com/automation/notifications/slack.md), and incidents in [ServiceNow](https://docs.germainux.com/automation/servicenow.md)

## Set It Up

Each layer adds value on its own. The more layers you monitor, the faster you rule out the ones that didn't change.

**1. Monitor each layer**

Deploy monitoring for the layers your releases touch, such as [MuleSoft](https://docs.germainux.com/integrations/mulesoft-monitoring.md), [NGINX](https://docs.germainux.com/integrations/nginx-monitoring.md) and [PostgreSQL](https://docs.germainux.com/integrations/postgresql-monitoring.md), and [RUM JS](https://docs.germainux.com/deploy/germain-ux-rum-js.md) for the applications your users work in.

**2. Show releases on the timeline**

[Configure KPI relationships](https://docs.germainux.com/analytics/kpis/kpi-relationships.md#configuration) from your error, response-time and resource KPIs to your deployment and change events, and show them as [markers](https://docs.germainux.com/analytics/kpis/kpi-relationships.md#marker).

**3. Test after each deployment**

[Record synthetic scenarios](https://docs.germainux.com/automation/user-click-synthetic-user-scenario/record-a-synthetic-scenario.md) and [HTTP checks](https://docs.germainux.com/automation/synthetic-http-execution.md) for the paths a release must never break.

**4. Set SLAs and alerts**

Set [percent change SLAs](https://docs.germainux.com/analytics/slas.md#percent-change-sla) and [fixed SLAs](https://docs.germainux.com/analytics/slas/configure-sla.md) on the KPIs of each layer, and [alerts](https://docs.germainux.com/automation/notifications/create-an-alert.md) to the teams that own them.

## Additional Resources

> [card] [KPI Relationships](https://docs.germainux.com/analytics/kpis/kpi-relationships.md)
>
> Trends, markers and related KPIs

> [card] [Analysis Dashboard](https://docs.germainux.com/dashboards/analytics-dashboard/4-analysis-dashboard.md)
>
> Aggregate and instance analysis

> [card] [Use Cases](https://docs.germainux.com/use-cases/overview.md)
>
> More Use Cases

## Get Help

Every team ships releases differently, through its own pipeline and to its own layers. The GermainUX team can help you bring your deployments onto the timeline, choose the SLAs that catch regressions early and set up the checks that run after each release.

> [card] [Contact GermainUX Support](https://docs.germainux.com/getting-started/support.md)
>
> The GermainUX team helps you get up and running faster, empower your team, turn data into actionable insights, and optimize performance to deliver measurable business outcomes.

> [card] Can't Find What You Need?
>
> Search all GermainUX docs with **Ctrl + K**, or use the feedback buttons at the bottom of this page to tell us what is missing.

The content of this document is proprietary to GERMAIN SOFTWARE.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.