Skip to main content
ComponentsEngine

What GermainUX Watches

Errors, speed, resources and releases

Find the Cause

Three worked examples

Alert and Act

Catch the next one sooner

Set It Up

Layers, markers and tests

Overview

Something went wrong in production, and a release went out this morning. The first question in the incident call is always the same: did the release cause it, and should we roll back? Developers, operations and database teams each check their own layer, and the release that caused it may not be the one everybody suspects. A health check answers only part of it. A bad release can fail half the calls of one API, slow down two pages without a single error, or leak resources for hours before anything breaks. Each one shows on a different layer and at a different time. GermainUX puts the releases on the same timeline as the KPIs of every layer. It compares the period after the release with the period before, shows which API, endpoint or host changed, and lets you rule out what stayed normal. You decide to roll back or fix forward from evidence, and then confirm that it worked.
AlertAggregate AnalysisRelease MarkerAffected LayerRoll Back or FixConfirm
This page uses a MuleSoft integration layer, NGINX web servers and a PostgreSQL database as the example. The same approach works for any stack GermainUX monitors, and for releases that affect users directly, such as the theme release that slowed a Shopify product page.

Questions It Answers

  • Did this morning’s release cause the problem?
  • Should we roll back, or fix forward?
  • Which layer changed: the API, the web servers or the database?
  • How many orders or users did the release affect?
  • Is a release slowing things down without raising any error?
  • Did the rollback bring everything back to normal?

What GermainUX Watches

GermainUX keeps each signal as a KPI, so the analysis can compare the time after a release with the time before.

Find the Cause

The investigation follows the same path for every release:
  1. Open Aggregate Analysis on the KPI that raised the alert, from before the release to now. GermainUX compares the period with its baseline.
  2. Find the release on the Relationships chart. A marker, or the first change in a KPI, shows when it started.
  3. Read the Leading Factors to see which API, operation, route or host changed, and check the Related KPIs on the other layers.
  4. Rule out what stayed normal. It tells you whether the release is the only suspect.
  5. Roll back or fix, then run the same analysis on the period after.
The three examples below come from simulated data, and their numbers belong to the example. Each release breaks a different layer, in a different way.

Example: A Payments API Release Fails Almost Half the Payments

At 08:00, a new version of the payments API rolls out on MuleSoft. Five minutes later, payments start to fail. In this example, 45% of payment calls fail, against less than 1% before.
1

Spot the failures

Open the analysis of the MuleSoft API transactions for the payments API. Failures of POST /api/payments jump at 08:05, with a new error type, MULE:EXPRESSION. Response time doesn’t change, so an alert on response time alone wouldn’t fire.
2

Find the release

The release marker of the payments API sits at 08:00. Its availability shows one of its two replicas starting at 08:00 and 08:05: the rollout of the new version. The failures begin as the new replicas take traffic. Hover over the marker to see what the release changed. In this example, version 1.4.2 of the payments API changed the payment flow and its card mapping.
Payment Failures analysis of payments-papi where failed calls jump right after the release marker at 08:00 and fall back after the rollback marker at 09:15, with POST /api/payments ranked first in Leading Factors

Payment Failures After the Payments API Release - GermainUX

Release marker at 08:00 on the Payment Failures chart, its tooltip showing Deployment Run DEP-9412, payments-papi 1.4.2, completed on cloudhub-prod, with the changed files post-payment.xml, card-mapping.dwl and pom.xml

Payments API Release Details on the Release Marker - GermainUX

3

Measure the impact

Add the APIs that call it. The orders API and the EDI gateway now fail with 502 on the calls that need a payment, so about 45% of order placements fail too.
Analysis of MuleSoft calls that returned 502 after the payments API release, with Leading Factors ranking orders-api and POST /api/orders first, followed by edi-gateway

Calls Failing With 502 After the Payments API Release - GermainUX

4

Rule out the rest

The customer and inventory APIs, reading a payment, and the runtime’s JVM metrics all stay at their baseline. Only the new version’s payment call changed.
Roll back. The replicas show the rollback starting at 09:15, and failures are back below 1% by 09:20. The error type tells the developers where to look before the next attempt.

Example: A Storefront Release Makes Search Four Times Slower

At 11:00, a storefront release changes the search and product-review pages. Nothing fails: every request still returns 200. But in this example, search takes about 1 second instead of 0.3, and reviews 0.7 seconds instead of 0.18.
1

Spot the slowdown

Open the analysis of the inbound HTTP requests of the storefront for the day. Response time rises from 11:00 and stays high, while errors and traffic stay flat.
2

Find the release

The storefront’s release marker sits two minutes before the slowdown, and both storefront servers log a configuration reload at the same time. Hover over the marker to see what the release changed. In this example, it changed the search and review scripts and the server configuration.
Search Response Time analysis of the storefront where the average response time of /search rises at the release marker at 11:00 and stays high for the afternoon

Search Response Time After the Storefront Release - GermainUX

Release marker at 10:58 on the Search Response Time chart, its tooltip showing Web Release DEP-9377, storefront search and reviews, completed on web-01, with the changed files storefront.conf, results.js, facets.js and widget.js

Storefront Release Details on the Release Marker - GermainUX

3

Read what changed

Filter the Segmentation Breakdown to the HTTP paths. In this example, /search averages 1.1 seconds and stands out as an outlier, while the other storefront routes stay below 0.2 seconds. The product-review API slowed down too.
Storefront Response Time analysis with the Segmentation Breakdown filtered to HTTP paths, where the /search segment is outlined in red, averaging 1.141 seconds and flagged as an outlier, while the other routes stay below 0.16 seconds

Search Stands Out Among the Storefront Routes - GermainUX

4

Rule out the rest

The customer portal on the third server, the other storefront routes and the static files stay at their baseline. The servers aren’t overloaded; two routes changed.
Revert or fix the search and review changes, then compare the response time of the two routes with the period before the release.

Example: A Release Leaks Database Connections for Three Hours

At 06:45, an application release goes out. Nothing fails for more than two hours. Then, at about 09:20, the database starts refusing new clients.
1

Spot the outage

Database availability on the primary drops to 0, with FATAL: sorry, too many clients already. While the database refuses clients, GermainUX gets no connection counts, so the database connections KPI has a gap for that period.
Database Availability drillthrough of pg-prod-01 where every reading from 09:20 to 09:55 is 0 and readings return to 100 from 10:00

Primary Database Availability Drops to 0 - GermainUX

2

Look before the outage

Widen the analysis of database connections to the whole morning. In this example, they normally follow traffic, from about 120 in the morning to 260 at 10:00. From 06:45, they climb steadily, by 5 to 6 a minute, whatever the traffic. They pass 1,000 at about 09:05, and the connections SLA is breached from 09:10, 10 minutes before the outage.
3

Find the release

The climb starts at 06:45, exactly at the marker of the order service’s release, more than two hours before anything failed. Without the marker, you’d look for the cause around the outage, not at the start of the climb. Hover over the marker to see what the release changed. In this example, version 3.12.0 of the order service changed its order repository and its application configuration.
Database Connections analysis of pg-prod-01 where connections climb from the release marker at 06:45 to about 1,000 before the outage

Database Connections Climbing From the Order Service Release - GermainUX

Release marker at 06:45 on the Database Connections chart, its tooltip showing Deployment Run DEP-9341, orders-service 3.12.0, completed on app-prod-01, with the changed files OrderRepository.java and application.yml

Order Service Release Details on the Release Marker - GermainUX

4

Rule out the rest

The standby, deadlocks, dead rows and I/O stay at their baseline. It isn’t load or a slow query; connections are opened and never returned.
Restarting the connection pool brings the database back at 10:00, but only buys time. Roll back or fix the release, then check that connections follow traffic again.

Alert and Act

Each example points to a signal you can catch sooner next time:

Set It Up

Each layer adds value on its own. The more layers you monitor, the faster you rule out the ones that didn’t change.
1

Monitor each layer

Deploy monitoring for the layers your releases touch, such as MuleSoft, NGINX and PostgreSQL, and RUM JS for the applications your users work in.
2

Show releases on the timeline

Configure KPI relationships from your error, response-time and resource KPIs to your deployment and change events, and show them as markers.
3

Test after each deployment

Record synthetic scenarios and HTTP checks for the paths a release must never break.
4

Set SLAs and alerts

Set percent change SLAs and fixed SLAs on the KPIs of each layer, and alerts to the teams that own them.

Additional Resources

KPI Relationships

Trends, markers and related KPIs

Analysis Dashboard

Aggregate and instance analysis

Use Cases

More Use Cases

Get Help

Every team ships releases differently, through its own pipeline and to its own layers. The GermainUX team can help you bring your deployments onto the timeline, choose the SLAs that catch regressions early and set up the checks that run after each release.

Contact GermainUX Support

The GermainUX team helps you get up and running faster, empower your team, turn data into actionable insights, and optimize performance to deliver measurable business outcomes.

Can't Find What You Need?

Search all GermainUX docs with Ctrl + K, or use the feedback buttons at the bottom of this page to tell us what is missing.

The content of this document is proprietary to GERMAIN SOFTWARE.