Averages Vs Percentiles
Features
Watch out what you wish for when looking at averages when analyzing performance metrics. You will either miss issues or be misled about issues. Instead, rely on percentiles (p50, p90, p99, etc).
When analyzing performance metrics, relying solely on averages can be problematic. Averages can mask underlying issues in your data and provide an overly simplistic view of performance. For example, if you're looking at the average page load time across a large number of users, it might appear acceptable at first glance. However, this average can be heavily influenced by outliers—either users who experience very fast load times or those with extremely slow connections. These outliers can skew the average, making it seem like the overall performance is better than it actually is.
Why Averages Can Be Misleading:
Outliers Impact: Averages are susceptible to extreme values. A small number of users experiencing exceptionally poor performance can drag the average down, giving you a false sense of the system's performance for the majority of users.
Missed Insights: With averages, you might overlook critical issues that only affect a subset of users. For example, if most users experience fast load times, but a small group faces delays, the average will smooth over the problem, and you may fail to identify and address the root cause.
Why Percentiles Are Better:
Percentiles offer a more granular and accurate view of performance, especially in datasets with a wide range of values or outliers. By using percentiles, such as the 50th percentile (p50), 90th percentile (p90), and 99th percentile (p99), you can understand how the majority of users are experiencing your service, as well as the tail ends of your performance spectrum.
p50 (Median): The p50 shows the median value of the dataset, meaning half of the users experience performance better than this, and half experience worse. This gives you a solid idea of what the typical user is experiencing.
p90 (90th percentile): This value tells you that 90% of users are experiencing performance better than this point. It helps you understand how your system is performing for the majority of users, especially those who might be at the higher end of the performance scale.
p99 (99th percentile): This represents the worst-case performance for 1% of users, identifying those who experience the slowest performance. By focusing on p99, you can spot and resolve issues that impact a small but critical subset of users.
By focusing on percentiles, you gain a more complete picture of performance across different user groups, allowing you to identify both the general trends and specific issues affecting performance. This can help ensure that you address problems that might otherwise be overlooked with averages, especially for users experiencing poor performance.
In summary, while averages can provide a quick snapshot, percentiles offer much deeper insights into the true performance of your system, helping you avoid being misled and ensuring that you address issues affecting the full spectrum of your users.
Video explaining the difference: https://youtu.be/Y-mY97qoeyw
Check out GermainUX’s various duration Measures.
Service: Analytics
Feature Availability: 2017.3