8.How to measure an app growth experiment

In this chapter: how to decide, before you change anything, what you will measure, for whom, for how long and against what — and how to reach an honest verdict afterwards, including "inconclusive". Correlation, causation and small samples explained in plain language, with every rule of thumb's assumptions stated.

The problem

You changed something. A number moved. It is very hard not to connect the two. And when the number moves the wrong way, it is just as hard not to find a reason why "it doesn't count".

Three different statements get mixed up here, and this chapter is about keeping them apart:

1. The metric changed. Downloads per week went from 400 to 460.
2. The goal was reached. We wanted 450 per week; we are above it.
3. Our action caused it. Without the new screenshots, we would still be at 400.

The first is an observation. The second is a comparison with a target you set. The third is a claim about a world you did not observe — the one where you did nothing — and it needs the most evidence.

What is going on

Five things to write down before you change anything

Baseline. What the metric looked like before, over a period long enough to show its normal range. One week is rarely enough; 8–12 comparable periods are a practical starting range, not a universal sample-size rule. Seasonality, autocorrelation or structural changes can make that range insufficient.

Exposure. Who actually experiences the change. A new screenshot only affects people who see the product page or the search result that shows it. A custom product page only affects visitors who arrive through its URL or targeting [S03] [S11]. A change in a release only affects users on that version. Measure the metric for the exposed group where you can; the whole-app total dilutes the effect.

Window. When you will look, and for how long. Decide in advance. If you keep looking until the number looks good and stop there, you will almost always find a "win" — that is chance, not the change.

Confounders. Anything else that could move the metric in the same window: holidays and seasons, releases, price changes, featuring, ads, press or community posts, competitor launches, store outages — and definition changes. For example, Google Play's page dates the primary unique-click metric to June 2026 and the store-listing report update to July 2026, replacing the older conversion-rate headline with click-through rate [S12]; a before-and-after comparison across that change compares different metrics. Play still offers the older store listing acquisitions in Statistics and exports [S12], so a continuous series is available if you look for it.

Verdict rules. What result counts as "worked", "did not work", "harmful" and "inconclusive". Write them before you look.

Correlation and causation, in plain words

Two things that move together are correlated. That can happen because one causes the other, because something else causes both (a holiday increases both downloads and reviews), or by chance.

You get closer to causation by comparison: the same kind of people, in the same period, with and without the change. That is what product page optimization on the App Store [S04] and store listing experiments on Google Play [S10] do — they split traffic at the same time, so seasons and news affect both sides equally. When you cannot split traffic, look for a comparison group that did not get the change but faces the same conditions: another country where you did not change the page, or the other platform. Be careful: iOS and Android users differ, prices and offers differ, and store algorithms differ. The other platform is a helpful hint, never a true control group.

Small samples: how much can a number move on its own?

A rough check for counts (downloads, purchases, reviews in a period):

• Assumptions: events happen independently of each other, the underlying rate is stable over the periods you compare, and the counts are comfortably above 20. These are Poisson/normal sanity approximations, not significance tests; use exact Poisson/binomial intervals or longer windows for very small counts, clustered events or materially varying rates.
• Rule of thumb for one period: a single period's count usually falls within about ±2√N of its long-run average N, by chance alone.
• Rule of thumb for two periods: two periods with the same underlying rate can differ from each other by about ±3√N (more precisely ±2.8√N), because both of them wobble.
• Examples: 400 downloads a week → a week between 360 and 440 is unremarkable, and two weeks 55 apart are unremarkable. 25 purchases a week → one week within ±10, two weeks up to about 14 apart. 9 purchases a week → below the range where these rules are even reasonable; use longer periods.
• Better still: compare against your own 8–12-period normal range when the periods are comparable; it contains your app's real wobble, including effects the rule of thumb ignores.

For rates (conversion, trial-to-paid), report the estimated difference and any interval or precision information available, then use the planning rule in chapter 4 or the store's own test statistics. Apple reports a confidence level for product page optimization and suggests waiting for at least 90% [S04]; Google Play lets you set the confidence level and the minimum detectable effect, and declares a draw when the difference is smaller than that effect [S10]. Read these as the stores' own summaries under their own assumptions. Even a confident result does not tell you exactly how large the true effect is, or whether it will hold once the test ends and the season, traffic mix or competition changes. And a store's result covers only the metric it measures — usually downloads or clicks (Install, Open or Pre-register clicks in current Play reports), not paying users.

Regression to the mean

If you act right after an unusually bad week, the next week is likely to be better whatever you do — unusual weeks tend to be followed by more ordinary ones. This is why "we fixed it and it recovered" is weak evidence when the fix came right after a dip. A baseline of several periods helps you see whether the dip was unusual.

"Inconclusive" is a result

An inconclusive result means the prespecified decision threshold was not reached; nonsignificance alone does not show that the effect is small. Report the estimated effect and interval or precision information available, then state whether the planned power or precision was sufficient to exclude a meaningful effect. Only an interval that rules out your prespecified meaningful effect supports excluding that effect under the analysis assumptions. Planned power alone cannot do this after the result is known; a low-powered or repeatedly peeked test does not establish that the effect is small. Record it as such (chapter 11), and do not quietly relabel it as a win or a loss.

Worked example

Fictional example · E8 · Pantrio · tags: ios, android, conflicting-signals, inconclusive

Situation. Pantrio (the meal-planning app from chapter 5) replaced its iOS screenshots on 1 September with a new set focused on "plan a week in five minutes". The change was made directly, not as a split test. Android's listing was not changed. Goal: more downloads, and ultimately more paying users. Four weeks later, the developer wants to know whether it worked.

Measurement card (written, belatedly, on 29 September):

Observations (4 weeks after vs 8-week baseline, weekly averages):

• iOS conversion rate (App Store Connect [S05]): up from about 5.1% to about 5.7%.
• iOS unique-device impressions: down about 9%.
• iOS total downloads: roughly flat (+2%).
• Android downloads (listing unchanged): up about 10%.
• New paying users, both platforms: flat, about 30 per week on each platform, within the range of the baseline weeks (for a weekly count of 30, the rule of thumb gives about ±11 for a single week).

The signals conflict. Conversion on iOS rose, which fits the new screenshots working. But downloads did not rise, because fewer people saw the app. Android — with no change — rose more than iOS, which fits a seasonal lift in interest that iOS should also have received. If iOS had the same seasonal lift as Android, then the flat downloads could mean the new screenshots did worse; or the drop in iOS impressions may have hidden an improvement. Paying users did not move beyond normal variation on either platform.

Hypotheses.

1. The new screenshots improved iOS conversion. Possible; the conversion rise is consistent with it.
2. The conversion rise is a mix effect: fewer impressions from broad browse traffic (which converts less), leaving a larger share of high-intent search visitors. Possible; impressions fell mainly in browse (checked in the source view [S23]).
3. Seasonality lifted interest on both platforms, and iOS search visibility fell for unrelated reasons. Possible.

Decision: Inconclusive — keep the new screenshots, claim nothing. The change is cheap to keep and shows no sign of harm. The journal entry reads: "Metric changed: iOS CR up, downloads flat. Goal not reached. Causation: not established; seasonality and traffic mix confound." For the next listing change, the developer will use product page optimization [S04] so both versions face the same season, and will write the verdict rules first. Separately, the drop in iOS browse impressions goes to a discoverability check (chapter 3).

Do this now

1. Pick one change you made in the last three months and fill a measurement card for it, honestly, even if late.
2. Check your baseline range: list the metric for the 8–12 comparable periods before the change.
3. Write down the exposed group and whether you can measure it separately.
4. List confounders in the window, including definition changes in any console.
5. Apply the counts rule (with its assumptions) or the store's own test result.
6. Write a verdict: worked, did not work, harmful or inconclusive — and the three separate statements (metric changed / goal reached / caused by us).
7. For your next change, fill the card before you ship it.

Don't change anything yet when…

• You have not written the baseline, window and verdict rules.
• A known confounder (holiday, release, price change, featuring, campaign) falls inside the window and you cannot separate it.
• You are still inside the window of a previous change to the same step.
• Your counts are too small for the window you chose; lengthen the window before you start.
• The metric's definition changes during the window.

Worksheet

Measurement card — baseline, exposure, window, confounders, verdict rules, the counts check with its assumptions, and the three separate statements.

Optional: each move Next Move proposes includes a measurement instruction — what to watch, over what window, and its limits. Indevaro is being built to record outcomes against that instruction in the full product; today, this card is how you do it.

Sources

• S03 — Custom product pages (Apple Developer) — https://developer.apple.com/app-store/custom-product-pages/
• S04 — Product page optimization (Apple Developer) — https://developer.apple.com/app-store/product-page-optimization/
• S05 — Metric definitions, App Store Connect Analytics (Apple Developer Help) — https://developer.apple.com/help/app-store-connect-analytics/reference/metrics-definitions/
• S10 — Store listing experiments, Play Console Help — https://support.google.com/googleplay/android-developer/answer/12053285
• S11 — Custom store listings, Play Console Help — https://support.google.com/googleplay/android-developer/answer/9867158
• S12 — Acquisition reports, Play Console Help — https://support.google.com/googleplay/android-developer/answer/9859173
• S23 — Acquisition, App Store Connect Analytics (Apple Developer Help) — https://developer.apple.com/help/app-store-connect-analytics/acquisition/acquisition/