11.How to learn from app growth experiments

In this chapter: how to write down what you decided, what you actually did, and what happened — and then revise what you believe about your app and its users. A short decision journal turns scattered experiments into knowledge you can use in the next decision.

The problem

Six months into running an app, most developers can list what they changed. Very few can say what they learned. The screenshot test from spring? "I think it helped." The forum post? "Some people came." The price change? "Revenue went up, but so did downloads, so who knows."

Without a record, three things happen. You repeat tests you already ran. You keep beliefs that your own evidence contradicted. And your judgment never gets better, because you never compare what you expected with what you got.

What is going on

Four records, not one

A useful journal entry separates four things that are easy to blur:

1. The decision. What you chose, the evidence behind it, the alternatives you rejected and why, and what you expected to happen. Written before acting. This is your decision card (chapter 1), filed.
2. The execution. What you actually did, when, and how it differed from the plan. Plans change: the test ran at 50% instead of 30%, the post went out on a Tuesday instead of a Saturday, the fix shipped with an extra change. Record it.
3. The outcome. What the measurement card (chapter 8) showed, with the three statements kept separate: the metric changed or not; the goal was reached or not; the evidence for causation is strong, weak or absent. "Inconclusive" is an outcome.
4. The learning. What you now believe, how strongly, and what would change your mind.

Beliefs, stated so they can be wrong

A learning is a belief about your app and its users, written so that future evidence could contradict it. Compare:

• Vague: "Users like simple screenshots."
• Useful: "For Coinmoth's US search visitors, a first screenshot about setup speed converts better than one about spending history. Confidence: moderate (one test, one market). Would revise if a test in another market shows no difference, or if the effect disappears after the new-year season."

Three habits keep beliefs honest:

• Write the prior. Before the outcome, write what you believed and how strongly. Afterwards, compare. If every result "confirms" what you thought, you are probably not writing priors.
• Move in proportion to the evidence. One inconclusive test should barely move a belief. A clean split test with a clear result should move it a lot. A before-and-after change during a holiday should move it very little.
• Keep negative and null results. "Custom page for Reddit: 9 visitors, no downloads" is valuable. It stops you, or a collaborator, from trying the same thing next quarter under a new name.

Review on a schedule

Once a month, read the last month's entries and ask:

• Which beliefs changed? Which should now change the default of what you do?
• Which decisions were right for the wrong reasons, or wrong for good reasons? (A good decision can have a bad outcome; judge the reasoning with the information available at the time.)
• Which open questions keep coming back? They are candidates for a proper test.

Keep it small

A journal nobody keeps is worse than a short one. One entry per decision, a few lines per section. A spreadsheet or a plain text file is enough.

Worked example

Fictional example · E11 · Coinmoth · tags: ios, learning, listing-test

Situation. Coinmoth (chapter 4) re-ran its product page optimization test with one treatment and 50% of traffic. This example shows the full journal entry — decision, execution, outcome and learning — and the belief revision that followed.

Decision (written 6 October):

• Chose: test "Set up a budget in 60 seconds" on the first two screenshots against the original, US, one treatment, 50% of traffic. Minimum run 7 full weeks (the planned ~9,600 visitors per arm); after that, apply if Apple reports the treatment better at its suggested confidence [S04]; stop at 90 days as inconclusive otherwise.
• Evidence: the previous test was inconclusive (too small); review themes mention quick setup; the treatment message is true of the current version.
• Rejected: the color-scheme variant (no written hypothesis).
• Prior belief: "Setup-speed messaging converts better for search visitors." Confidence: low.
• Expected: conversion up by about 0.8 points, if the idea is right; otherwise no difference.

Execution:

• Started 9 October, three days late, because App Review had questions about the new screenshots [S04].
• No other listing changes in the US during the test. A minor bug-fix release shipped on day 20 — noted as a possible confounder; it did not touch onboarding.

Outcome (measurement card, 27 November, day 49 — the end of the minimum run):

• The treatment was reported as performing better than the original at Apple's suggested confidence level [S04]. Observed conversion: about 4.1% original vs about 4.8% treatment.
• Metric changed: yes, for exposed visitors in the test. Goal (more US downloads): total US downloads rose about 7% over the test period, within a season that also lifts budgeting apps — so the goal was only partly reached, and the rise is not all attributable to the test.
• Causation: reasonably strong for conversion (simultaneous split test), not established for total downloads.

Decision taken on the outcome: apply the treatment as the default product page; keep the measurement card open for four more weeks to see whether the conversion difference holds outside the test.

Learning:

• Updated belief: "For Coinmoth's US search visitors, setup-speed messaging converts better than spending-history messaging." Confidence: moderate (one clean test, one market, one season).
• Would revise if: conversion falls back to the old level within four weeks after applying, or a UK test shows no difference.
• Next use: the same message goes into the custom product page for the next community post (chapter 9), as a separate, measured experiment.
• Also learned: the first test's "inconclusive" was a sample problem, not a message problem — recorded against the first entry, so it is not read later as "the message failed".

Decision: Apply the tested page, keep watching for four weeks, raise confidence in the belief from low to moderate — not to high.

Do this now

1. Create a journal — a spreadsheet or text file — with four sections per entry: decision, execution, outcome, learning.
2. Back-fill your last three decisions from memory and data, marking anything you are unsure of as "not recorded".
3. For your next decision, write the prior belief and the expected result before acting.
4. When the measurement window closes, write the outcome with the three separate statements.
5. Write the learning as a belief with a confidence level and a "would revise if".
6. Put a monthly review in your calendar: 30 minutes to read the month's entries.

Don't change anything yet when…

• You have not written down what you expect to happen — you will not be able to tell whether the outcome surprised you.
• The previous change to the same step has no recorded outcome; close that entry first.
• A single result would flip a belief that several earlier results supported; look for a confounder before revising strongly.
• The "learning" you want to record is really a hope ("it will probably work next time").

Worksheet

Decision journal — decision, execution, outcome (three statements) and learning (belief, confidence, would revise if), plus a monthly review page.

Optional: the full Indevaro product is being built to keep this record for you — linking a decision to what was done, the measured result and the lesson used in the next recommendation. Today, the free Next Move tool gives you a move and a measurement instruction to start an entry; the journal is yours.

Sources

• S04 — Product page optimization (Apple Developer) — https://developer.apple.com/app-store/product-page-optimization/