The Perils of Chasing p99. Hidden correlations can mislead… | by Krishna Rao | Jun, 2024 – Niraranra

p99, or the price beneath which 99% of observations fall, is extensively used to hint and optimize worst-case effectivity all through industries. For example, the time taken for an internet web page to load, fulfill a shopping for order or ship a cargo can all be optimized by monitoring p99.

Whereas p99 is undoubtedly priceless, it’s important to acknowledge that it ignores the very best 1% of observations, which may have an unexpectedly large impression once they’re correlated with completely different essential enterprise metrics. Blindly chasing p99 with out checking for such correlations can doubtlessly undermine completely different enterprise targets.

On this text, we’re going to analyze the constraints of p99 by the use of an occasion with dummy information, understand when to rely on p99, and uncover alternate metrics.

Take into consideration an e-commerce platform the place a workforce is tasked with optimizing the shopping for cart checkout experience. The workforce has acquired purchaser complaints that testing is pretty sluggish compared with completely different platforms. So, the workforce grabs the newest 1,000 checkouts and analyzes the time taken for testing. (I created some dummy information for this, you is perhaps free to utilize it and tinker with it with out restrictions)

import pandas as pd
import seaborn as sns
order_time = pd.read_csv('https://gist.githubusercontent.com/kkraoj/77bd8332e3155ed42a2a031ce63d8903/raw/458a67d3ebe5b649ec030b8cd21a8300d8952b2c/order_time.csv')
fig, ax = plt.subplots(figsize=(4,2))
sns.histplot(information = order_time, x = 'fulfillment_time_seconds', bins = 40, coloration = 'okay', ax = ax)
print(f'p99 for fulfillment_time_seconds: {order_time.fulfillment_time_seconds.quantile(0.99):0.2f} s')

As anticipated, most shopping for cart checkouts look like ending inside only a few seconds. And 99% of the checkouts happen inside 12.1 seconds. In numerous phrases, the p99 is 12.1 seconds. There are only a few long-tail cases that take as long as 30 seconds. Since they’re so few, they is perhaps outliers and must be protected to ignore, correct?

Now, if we don’t pause and analyze the implication of the ultimate sentence, it may presumably be pretty dangerous. Is it really protected to ignore the very best 1%? Are we sure checkout cases is not going to be correlated with another enterprise metric?

Let’s say our e-commerce agency moreover cares about gross merchandise price (GMV) and has an basic company-level objective to increase it. We should always all the time immediately confirm whether or not or not the time taken to checkout is correlated with GMV sooner than we ignore the very best 1%.

import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter
order_value = pd.read_csv('https://gist.githubusercontent.com/kkraoj/df53cac7965e340356d6d8c0ce24cd2d/raw/8f4a30db82611a4a38a90098f924300fd56ec6ca/order_value.csv')
df = pd.merge(order_time, order_value, on='order_id')
fig, ax = plt.subplots(figsize=(4,4))  
sns.scatterplot(information=df, x="fulfillment_time_seconds", y="order_value_usd", coloration = 'okay')
plt.yscale('log')
ax.yaxis.set_major_formatter(ScalarFormatter())

Oh boy! Not solely is the cart price correlated with checkout cases, it should enhance exponentially for longer checkout cases. What’s the penalty of ignoring the very best 1% of checkout cases?

pct_revenue_ignored = df2.loc[df1.fulfilment_time_seconds>df1.fulfilment_time_seconds.quantile(0.99), 'order_value_usd'].sum()/df2.order_value_usd.sum()*100
print(f'If we solely focussed on p99, we'd ignore {pct_revenue_ignored:0.0f}% of revenue')
## >>> If we solely focussed on p99, we'd ignore 27% of revenue

If we solely centered on p99, we’d ignore 27% of revenue (27 cases greater than the 1% we thought we had been ignoring). That’s, p99 of checkout cases is p73 of revenue. Specializing in p99 on this case inadvertently harms the enterprise. It ignores the needs of our highest-value prospects.

df.sort_values('fulfillment_time_seconds', inplace = True)
dfc = df.cumsum()/df.cumsum().max() # % cumulative sum
fig, ax = plt.subplots(figsize=(4,4))
ax.plot(dfc.fulfillment_time_seconds.values, coloration = 'okay')
ax2 = ax.twinx()
ax2.plot(dfc.order_value_usd.values, coloration = 'magenta')
ax.set_ylabel('cumulative achievement time')
ax.set_xlabel('orders sorted by achievement time')
ax2.set_ylabel('cumulative order price', coloration = 'magenta')
ax.axvline(0.99*1000, linestyle="--", coloration = 'okay')
ax.annotate('99% of orders', xy = (970,0.05), ha="correct")
ax.axhline(0.73, linestyle="--", coloration = 'magenta')
ax.annotate('73% of revenue', xy = (0,0.75), coloration = 'magenta')

Above, we see why there’s a large mismatch between the percentiles of checkout cases and GMV. The GMV curve rises sharply near the 99th percentile of orders, ensuing inside the excessive 1% of orders having an outsize impression on GMV.

This isn’t merely an artifact of our dummy information. Such extreme correlations are sadly commonplace. For example, the very best 1% of Slack’s purchasers account for 50% of revenue. About 12% of UPS’s revenue comes from just 1 customer (Amazon).

To steer clear of the pitfalls of optimizing for p99 alone, we’ll take a further holistic technique.

One reply is to hint every p99 and p100 (the utmost price) concurrently. This fashion, we acquired’t be liable to disregard high-value prospects.

One different reply is to utilize revenue-weighted p99 (or weighted by gross merchandise price, income, or another enterprise metrics of curiosity), which assigns greater significance to observations with bigger associated revenue. This metric ensures that optimization efforts prioritize most likely probably the most priceless transactions or processes, pretty than treating all observations equally.

Lastly, when extreme correlations exist between the effectivity and enterprise metrics, a further stringent p99.5 or p99.9 can mitigate the prospect of ignoring high-value prospects.

It’s tempting to rely solely on metrics like p99 for optimization efforts. Nonetheless, as we observed, ignoring the very best 1% of observations can negatively impression a giant proportion of various enterprise outcomes. Monitoring every p99 and p100 or using revenue-weighted p99 can current a further full view and mitigate the hazards of optimizing for p99 alone. On the very least, let’s consider to steer clear of narrowly specializing in some effectivity metric whereas shedding sight of basic purchaser outcomes.