Chapter 6 of 12
Did an app update hurt growth? Check crashes and ANRs
In this chapter: how to connect three kinds of evidence — a release, error data and a business metric — before you blame marketing for a drop. You will learn how phased and staged rollouts limit the damage, and how to decide whether a fix comes before everything else.
The problem
Growth problems and technical problems look the same on a revenue chart. A crash on launch for one device family, a sign-in failure after an SDK update, a purchase flow that hangs on older OS versions: each shows up as "fewer active users" or "fewer paying users", often days before anyone reads the crash report.
Meanwhile, the usual growth reflexes make it worse. A new screenshot test started during a crash spike will be judged on broken traffic. A price change shipped in the same release will be blamed for — or credited with — whatever the bug did.
What is going on
Three kinds of evidence, one timeline
Put these on the same timeline, with dates and versions:
- Release evidence. Version number, release date, rollout percentage over time, what changed (including SDK and dependency updates, server changes, and remote configuration).
- Error evidence. Crash rate, freeze or "not responding" rate, error logs, failed purchases, failed sign-ins — by app version, OS version and device.
- Business evidence. Active users, sessions, trials, purchases, ratings — by the same version and device splits if you can.
A regression is plausible when the error evidence changes at the release date, is concentrated in the new version, and the business metric moves in the same segment. It is likely when rolling back or halting the release (or fixing the bug) reverses the business change in that segment. Timing alone is correlation; the segment match is what makes it convincing.
What the platforms give you
Google Play — Android vitals. Play collects stability and performance data from devices whose users allow it. Its core vitals include the user-perceived crash rate and the user-perceived ANR ("application not responding") rate. Play publishes bad-behavior thresholds — 1.09% for user-perceived crashes and 0.47% for user-perceived ANRs overall, and 8% for either on the app's aggregate for a single phone model — and says that above a threshold it may reduce your app's visibility and may show users a warning on your listing [S18]. Treat these as the platform's enforcement lines, not as targets: many regressions hurt users long before they cross them.
Google Play — staged rollouts. You choose the percentage of users who get an update; it does not increase on its own. If you discover a problem, halting the rollout stops any additional users from receiving the update; users who already have it keep it. A new release started before the previous rollout completed goes to the same group of users, depending on its percentage [S19]. If the fix must reach the same affected cohort, consider matching or exceeding the halted percentage after checking the current rollout controls; the source does not make that an absolute minimum.
App Store — phased release. An update can go to users with automatic updates over seven days: 1%, 2%, 5%, 10%, 20%, 50%, then 100%. You can pause for up to 30 days in total, and anyone can still update manually from the store during a phased release [S08].
App Store Connect — crashes. App Store Connect reports crash counts; usage-based figures come from users who opted in to share data with developers [S05]. For detail by device and stack trace, most teams also use a crash-reporting SDK.
Why the rollout controls matter for growth decisions
A staged or phased release is not only a safety net. It is a natural comparison: for a few days, some users run the new version and some the old one, and the platform picks the updated group at random [S19] [S08]. The comparison is cleaner on Google Play; on the App Store the random part covers only users with automatic updates, while anyone who downloads or updates manually gets the new version [S08], so those users are self-selected. If errors and business metrics differ between versions in that window, you have much better evidence than a before-and-after chart. Use the window: look at the numbers by version on day 2 or 3, not only after 100%.
Ranking a fix against growth work
A confirmed regression that affects a meaningful share of users usually comes first, for three reasons: it hurts users now, it corrupts every other measurement you are running, and on Google Play it can affect visibility if it crosses the bad-behavior thresholds [S18]. But "confirmed" matters. An unconfirmed error spike gets an investigation, not a week-long rewrite.
Worked example
Fictional example · E6 · Tallowbird · tags: android, technical-bug
Situation. Tallowbird is a habit tracker for Android, built by a solo developer. Version 4.2 moved reminder scheduling to a new background library. It went out as a staged rollout at 20%. Three days later, the developer notices fewer trial starts on Android and plans to test a new paywall headline.
Timeline and evidence:
| Date | Release | Error evidence (Play Console, Android vitals) | Business evidence (own events) |
|---|---|---|---|
| Day 0 | 4.2 at 20% | — | — |
| Day 1 | 4.2 at 20% | User-perceived ANR rate for 4.2 noticeably higher than for 4.1; concentrated in one manufacturer's devices running one Android version | — |
| Day 3 | 4.2 at 20% | Same pattern; overall app ANR rate still under Play's 0.47% threshold, but the 4.2 rate on the affected device model is far above 8%. Play assesses the 8% per-phone-model threshold on the app/device-model aggregate over the last 28 days, so this version slice is evidence of risk, not itself a threshold crossing [S18] | Trial starts per active user on 4.2: clearly lower than on 4.1 in the same days |
| Day 3 | — | Six new 1-star reviews mention "freezes when I open reminders" | — |
Hypotheses.
- The new background library causes ANRs on one device family, which stops users before they reach the reminder setup where the trial offer appears. Strong: timing, version and device segment match, and reviews describe the symptom.
- The paywall headline is weak. Weak: 4.1 users in the same days, with the same paywall, started trials at the usual rate.
- Noise. Unlikely: the difference is concentrated in exactly the segment with the errors.
Decision: Halt the rollout and fix first. The developer halted the 4.2 staged rollout so no additional users receive it [S19], reverted reminder scheduling for the affected OS version in 4.2.1, and released 4.2.1 as a staged rollout at 20% — matching the halted share after checking the current rollout controls. When a new release starts before the previous rollout is complete, Play uses the same group of users as the previous release, depending on the percentage [S19]; matching the share was the team's chosen way to target that affected cohort, not a source-mandated minimum. The check on day 2 is by version and device. The paywall headline test is postponed until two weeks after a stable release, because any result during the regression would mix two effects. Written in the decision journal: "Trial drop explained by 4.2 ANR on one device family; not a paywall signal."
Follow-up. On 4.2.1, the ANR rate for the affected device family returned to the 4.1 level and trial starts by version matched again. That is the reversal that turns "plausible" into "likely".
Do this now
- Build a release timeline for the last 90 days: version, date, rollout percentage, what changed (including SDKs, server and remote config).
- Next to each release, add error evidence by version: crash rate, ANR or freeze rate, failed purchases, failed sign-ins.
- Add one business metric by version: trial starts, purchases or day-1 return rate from your own events.
- Look for a segment match: a version, OS or device where errors and the business metric moved together.
- Set a rollout habit: staged rollout on Play [S19], phased release on the App Store [S08], with a written "halt if" rule and a day-2 check by version.
- Postpone growth tests that would overlap an unresolved regression.
Don't change anything yet when…
- You have an error spike but no version or device concentration — investigate before you rewrite.
- The business metric moved in all versions equally; the release is probably not the cause.
- A growth test is scheduled during an unresolved regression — postpone it.
- You are tempted to ship a fix and a new feature together; ship the fix alone so you can see its effect.
- Your only error evidence is a handful of reviews; they justify an investigation, not a conclusion.
Worksheet
Release regression triage — release, error evidence and business evidence by version and segment, with a halt-or-proceed rule and a fix-priority check.
Optional: Next Move uses public data only, so it cannot see your crash or error data; when public reviews hint at a technical problem, it can propose an investigation checklist, never a diagnosis. Indevaro is being built to connect release and error sources with business metrics in the full product; that capability is planned, not part of the free tool.
Sources
- S05 — Metric definitions, App Store Connect Analytics (Apple Developer Help) — https://developer.apple.com/help/app-store-connect-analytics/reference/metrics-definitions/
- S08 — Release a version update in phases (App Store Connect Help) — https://developer.apple.com/help/app-store-connect/update-your-app/release-a-version-update-in-phases/
- S18 — Android vitals (Android Developers) — https://developer.android.com/google/play/vitals
- S19 — Staged rollouts, Play Console Help — https://support.google.com/googleplay/android-developer/answer/6346149
Worksheet for this chapter
Optional: try it on a real listing
The free Next Move form prepares one move from public evidence, with this chapter's goal already chosen. Nothing runs until you choose Generate, and this chapter works without it.
Find my next move — free