Every user got faster. The average got worse.

When a page gets much faster, people who used to give up on it start loading it. They tend to be on the slowest connections, so they can pull the average load time up even though every user got faster. Before you trust an average after a speed change, check whether the same people are in it.

The clearest case I know is YouTube’s Feather project. Around 2009, Chris Zacharias cut the watch page, the page that plays a video, from about 1.2 MB to 98 KB. He turned it on for a fraction of traffic and waited a week.

After a week of data collection, the numbers came back… and they were baffling.
The average aggregate page latency under Feather had actually INCREASED.

Chris Zacharias, Page Weight Matters, 2012

He wrote the story up in 2012. He published no overall numbers and no chart, so the next figure is a model that fits his words, not YouTube’s data.

He tested it in browsers repeatedly, and “nothing made sense.” He was close to giving up on Feather when a colleague found the answer. It was geography.

The usual explanation needs too many slow users

The new page drew “a disproportionate increase in traffic” from Southeast Asia, South America, Africa and “even remote regions of Siberia,” all with slow connections. The usual explanation is a change in the mix. More of the users who could already load the old page now come from slow places.

In the figure, 3 groups of users each load the 98 KB page 5 to 8 times faster. Press “Switch to the 98 KB page”. The average falls too, which contradicts his data.

Can a page get faster for everyone and slower on average?

Model built from Zacharias’s published facts, not YouTube data

This is the 1.2 MB page. Switch to the 98 KB page.

Slow region’s share of returning users’ loads1%

Average, all loads: 4.40 s. Slow region: 44 s. Slow share: 1.0%. The 98 KB page is not in use yet.

With JavaScript off: on the 98 KB page the fast region and the slow region’s returning users are 5 to 8 times faster, and the average still rises from 4.4 s to 6.6 s once 3,000 loads from new users in the slow region are counted.

Rings are the 1.2 MB page, dots the 98 KB page, and orange is users who could not load the old page. Set by hand, not by any published fact: the 4.0 s and 44 s starting times, an even 10% opt-in, the 5x and 8x speedups, and 3,000 new-user loads at 200 s. Log time axis; left is faster.

Now drag the slider to raise the slow region’s share of traffic. The average only passes its old value at 77%. This is Simpson’s paradox, where a trend in every group reverses in the total. A Hacker News commenter pointed it out at the time. But 77% in one week is hard to believe.

The new users had never been counted

In the slow regions, the average load under Feather took over 2 minutes. So the old page took over 20 minutes there. These are not more of the old users. They are people who could not load the old page at all, so the old average never included them. In his words, “Large numbers of people who were previously unable to use YouTube before were suddenly able to.”

Set the slider back to 1%, then press “Add new users from the slow region”. All 5 of his facts now hold, and every user is faster. The average goes from 4.4 s to 6.6 s. Almost all of the rise comes from new users, only 2.9% of loads. The code alone would have cut the average to 0.85 s.

Why the slowest users were missing

Real-user monitoring (RUM) collects timings from real browsers. Each page sends a small report, a beacon, once it reaches first paint, the first moment anything shows on screen. Someone who gives up before first paint sends nothing. So the slowest loads are missing by design.

In 2020, Nicolás Peña Moreno of Chrome’s speed metrics team measured how often slow loads are abandoned. He names this exact trap, where “a performance optimization causes more of the slower page loads to no longer be abandoned and hence could be perceived as a regression.”

What does RUM see when a page stops losing its slowest users?

Model: invented patience values, not measured

Cut the page size and watch the RUM average.

Reached first paint: 31 of 40. RUM average: 7.10 s. RUM p75: 5.17 s. True average over all attempts (only a model knows this): 105 s.

With JavaScript off: at 1.2 MB, 31 of 40 attempts reach first paint and RUM averages 7.10 s. At 98 KB all 40 do, every user is faster, and the RUM average rises to 9.25 s.

Grey dots are 30 users on fast connections. Orange marks are 10 users on slow ones. An orange ✕ is a user who gave up, drawn at the moment they left. Every slow user here gives up after 2 minutes, so the ✕ marks stack in one column. The patience values are invented: Peña Moreno’s data shows the mechanism, not these numbers.

Press the “Cut to” buttons in turn. Every user gets faster, and the loads that reach first paint go from 31 to 40. The RUM average still ends higher.

What to report after a launch

I once misread a speed number in the other direction. In 2023 I tried a tool that moves third-party scripts into a web worker, a background thread in the browser. The first measurements showed large gains. Then I found the scripts had never loaded. The page measured faster because the work was gone. I retracted the result and left notes for whoever picked it up next.

So when an average moves after a launch, check who is in it before you credit or blame the code. Then report 3 numbers instead of 1:

The average your users actually saw.
What the code did, with the old mix of users held fixed.
How many people you reached who could not load the page before. Count it from attempts in server logs.

The second number holds one thing fixed so a comparison shows one cause. In 2023 I did this for a slow market site, the company’s site for one South American country. Comparing the same image size there and in the US ruled out page weight. Users there still waited about 3 to 4 times as long, which pointed at the network.

In the model, the third number is 3,000 loads. Those are the users who nearly made Zacharias give up on Feather.