1.How to choose your next app growth experiment

In this chapter: you will learn to split every growth question into three parts — what you observed, what you suspect, and what you decide — and to write them on one card before you touch your listing, your price or your code. The card also tells you when the best decision is to change nothing yet.

Every example in this book is fictional. App names, numbers and outcomes are invented for teaching; any resemblance to a real app is unintended.

The problem

It is Monday morning. You open App Store Connect or Play Console and a number is lower than last week. Downloads are down. Or trials are flat while you expected them to rise. Within ten minutes you have a theory ("the new screenshots are worse", "a competitor launched", "the algorithm changed") and within an hour you are redesigning something.

Most indie developers do not lack ideas. They lack a pause between seeing a number and changing the product. Without that pause, three things go wrong:

1. You react to noise. Small apps have small numbers, and small numbers move a lot on their own.
2. You fix the wrong thing. A drop in downloads can start in discovery, on the product page, in a release, or in the calendar. Each needs a different response.
3. You cannot learn from what you did. If you changed three things in the same week, you will never know which one mattered.

What is going on

A useful growth decision has three layers. Keep them apart, in writing.

Observation (fact). Something you can point to, with its source, its time range and its definition. "Total downloads in the US App Store were 412 last week and 503 the week before, per App Store Connect." An observation has no because in it.

Hypothesis (a possible explanation). A statement that could be true or false and that some data could check. "Fewer people saw the app in search that week." "The new first screenshot converts worse." Always write at least two competing hypotheses, including a boring one ("normal week-to-week variation", "a holiday in the main market"). If you can only think of one explanation, you have not looked yet.

Decision (what you will do next). One of four kinds:

• Act — make a specific change, because the evidence is strong enough and the cost of being wrong is low.
• Test — make a change in a way that lets you compare (a store-listing experiment, a custom product page, a staged rollout).
• Check — gather the one missing piece of evidence that would decide between hypotheses.
• Wait — change nothing, set a date to look again, and write down what would make you act.

"Wait" and "check" are real decisions, not failures. They are often the best use of your time, because they protect you from undoing something that was working.

Where the numbers come from matters

Before you trust an observation, know how the platform defines it. Apple, for example, counts an impression when your app is seen for more than one second in the Today, Games, Apps or Search tabs, and its conversion rate is total downloads and pre-orders divided by unique-device impressions — which includes redownloads [S05]. Some App Store Connect usage figures only include users who agreed to share data with developers, and metrics are hidden until a minimum volume is reached [S05]. A number you misunderstand is not evidence; it is a guess wearing a label.

Three words to keep in your head

• Correlated means two things moved together. Downloads fell the same week you changed screenshots.
• Caused means one thing made the other happen. You can only say this when you have ruled out the other explanations or run a fair comparison.
• Noise is movement you would have seen even if nothing had changed. On a small app, noise is often larger than the effect of any single change.

This book returns to these words often. Chapter 8 shows how to measure an intervention without fooling yourself.

How strong is strong enough?

There is no universal threshold. Ask two questions:

1. How many independent kinds of evidence point the same way? A dip in downloads and a drop in impressions and a lost keyword position are three views of one story. Two dashboards showing the same underlying event are one kind of evidence, not two.
2. What does it cost to be wrong? Updating promotional text on the App Store can be done at any time without a new app version [S01]; being wrong costs little. Changing your price or your app name is harder to reverse and deserves more evidence.

Worked example

Fictional example · E1 · Hushlog · tags: ios, no-intervention

Situation. Hushlog is a small iOS sleep journal by a solo developer. Its goal this quarter is more downloads. On Monday the developer sees that total downloads in the United States were 412 last week, against 503 the week before — an 18% drop. The first screenshot was replaced two weeks ago, at the start of the week that had 503 downloads. The instinct is to revert it.

Observations (all from App Store Connect, US, same definitions):

• Total downloads: 503 → 412 week over week.
• Impressions (unique devices): down by a similar proportion, about 17%.
• Conversion rate (downloads ÷ unique-device impressions): roughly flat, within a few tenths of a point.
• The week before the screenshot change had 489 downloads; the first week with the new screenshot had 503.
• Last week included a long holiday weekend in the US.

Hypotheses:

1. The new screenshot converts worse. Weak: conversion did not drop; the loss is in how many people saw the app, which the screenshot does not affect much for people who never reach the page.
2. Fewer people searched for sleep-related apps last week (holiday, seasonality). Plausible, not proven.
3. Hushlog lost visibility in search for its main terms. Possible; needs a check of search-sourced impressions.
4. Normal variation. With about 400–500 downloads a week, a swing of this size is not unusual for this app (the developer's own last 12 weeks ranged from 398 to 540).

The trap. The screenshot change is recent and memorable, so it gets the blame. But the evidence points before the product page: fewer people saw the app, and those who saw it downloaded at the same rate.

Check performed (20 minutes). The developer opened the App Store Connect acquisition view, filtered to App Store search as the source type [S23], and compared the same two weeks. Search-sourced impressions fell by about the same share as total impressions; browse-sourced impressions fell too. No single source collapsed.

Decision: Wait. Keep the new screenshot. No listing change this week. Re-check in two weeks against the same weeks and against the 12-week range. Act only if (a) conversion rate falls below the 12-week low for two consecutive weeks, or (b) search impressions keep falling while browse impressions recover — which would point to a visibility problem worth a keyword check (chapter 3).

What the decision card looked like (abbreviated):

Do this now

1. Pick one number that worried you this week. Write it as an observation: metric name, value, comparison value, time range, market, source.
2. Look up its definition in the console's help for that metric. Note anything surprising (redownloads included? opt-in only? minimum thresholds?).
3. Write three hypotheses, including one boring one (calendar, noise, a definition change).
4. For each hypothesis, name one piece of data that would weaken it. If you already have that data, look now.
5. Choose a decision type: act, test, check or wait. If you choose act or test, write the smallest version of the change.
6. Set a re-check date and a trigger. "I will act if…" in one sentence.
7. Fill the decision card (worksheet below) and keep it where you will see it on the re-check date.

Don't change anything yet when…

• You cannot say which step of the funnel moved (chapter 2). Fix the diagnosis first.
• The change is within your app's own normal range over the last 8–12 comparable periods.
• The comparison window includes a holiday, a release, a price change, a featuring or an ad campaign that you have not accounted for.
• The metric's definition or reporting changed recently (chapter 2 describes a real metric change on Google Play).
• You changed something else in the same period and cannot separate the effects.
• The only evidence is one review, one tweet or one competitor's move.

Worksheet

Decision card — one page per decision: observation, hypotheses, evidence, decision type, re-check date and trigger.

Optional: if you want a second opinion based on your public listing, the free Next Move tool reads your public store listing (on the stores it supports) for one market and goal and returns one evidence-backed move — or a check, when the public evidence is not enough. The card above works without it.

Sources

• S01 — Creating your product page (Apple Developer) — https://developer.apple.com/app-store/product-page/
• S05 — Metric definitions, App Store Connect Analytics (Apple Developer Help) — https://developer.apple.com/help/app-store-connect-analytics/reference/metrics-definitions/
• S23 — Acquisition, App Store Connect Analytics (Apple Developer Help) — https://developer.apple.com/help/app-store-connect-analytics/acquisition/acquisition/