Group users by when they signed up or first acted, then watch each group's engagement curve, so a quiet quarter doesn't disguise a real shift in who is sticking
Aggregate engagement numbers move in ways that hide what is actually happening. A flat weekly active count can be the sum of older cohorts retaining well and new cohorts retaining badly, which looks fine until the older cohorts age out and the floor falls. A rising weekly active count can be the sum of a marketing push pulling in users who never come back, which looks like growth and is really churn delayed by a quarter. The aggregate is the average, and the average tells you nothing about which users are sticking.
Cohort analysis splits the population by when they entered (signup date, first-action date, first-purchase date) and tracks each group's engagement curve over weeks and months. The shape of the curve, and the difference between cohorts, is the actual signal: the question "are new users retaining as well as the cohort six months ago" has a yes-or-no answer once the cohorts are visible. The recipe is medium-low readiness because the data discipline is straightforward where event capture exists; the work is the cohort definition that matches how the product actually works and the freshness that keeps the curves trustworthy.
This recipe makes cohorts the unit of analysis rather than the aggregate, so engagement shifts surface as a change in a specific cohort's curve rather than as a vague move in the headline number.
A set of cohort retention curves that show how each entry group is engaging over time, with the comparison between cohorts as the read on whether the product is improving, holding, or quietly slipping. The metric is the cohort retention curve itself: the percentage of each cohort active at week one, week four, week twelve, and so on. The value shows up in catching real shifts early (a recent cohort retaining worse than older ones is a signal weeks before it shows up in the aggregate) and in answering the question that aggregate engagement cannot ("are users we acquired after the change retaining differently from users we acquired before").
The second outcome is the activation milestone tied to the cohort: identifying the in-product action that the retaining cohorts hit and the churning cohorts miss is what turns the curve from a diagnostic into an intervention target.
A warehouse with the event history, a cohorting query (most commonly written in SQL against the event store) and a visualization that makes the curves comparable. Composable stacks fit naturally because the warehouse is already the source of truth and product analytics tools (Amplitude, Mixpanel, PostHog) all run cohort analysis off the same shape of data. The capability that decides whether the analysis is honest is the cohort definition: a definition that drifts (sometimes signup, sometimes first action, sometimes first paid event) produces curves that mix apples and oranges and surface nothing useful, so the definition has to be set once, written down, and held stable across the period being analyzed.
Compare the tools on Martech Stack Builder
Analytics owns the cohort definitions, the retention curves, and the read of whether a between-cohort difference is real or noise. Product is critical because the activation milestone that anchors the curve is a product decision (what is the first action a user takes that signals they are likely to retain) and the interventions that come out of the analysis are product changes. Data engineering owns the event store, the cohort tables that materialise the analysis, and the freshness discipline that keeps the curves current rather than weeks stale. The recipe ships in weeks where event capture exists; longer where it does not, because the cohorts cannot be retrospectively reconstructed from data that was never collected. The activation-milestone work is iterative: the first cut of cohorts surfaces what the milestone is, and tuning it sharpens the curves over subsequent passes.
Work in this order. Lock the definition before reading any curve.
Memory and recall covers steps 2 and 4, behavioral segmentation steps 1, 3 and 7, and signal quality monitoring steps 5 and 6.
The first failure is the cohort definition that changes silently. If "signup-date cohort" sometimes means the calendar week and sometimes the rolling seven days, or "first-action" gets redefined when the activation milestone shifts, the curves stop being comparable across the period and the analysis loses its central use. The definition has to be fixed and the change recorded explicitly when it does shift, so a stair-step in the curve maps to a known redefinition rather than a phantom product event.
The second is missing the instrumentation regression. A cohort that drops sharply at week six is usually one of two things: a real product event that hurt that cohort, or an event-collection change that stopped firing the action the analysis depends on. Without a signal-quality check that catches volume anomalies in the underlying events, the team reads the second as the first and chases a problem that does not exist while a tracking bug runs unfixed.
The third is acting on cohort differences that are noise. Small cohorts produce noisy curves, and a five-percent week-four difference between two cohorts of two hundred users each is almost always within the bounds of natural variation. The discipline is sizing the cohorts large enough that real differences show up clearly, or using a longer window before declaring a trend, because a trend declared on a cohort too small to support it leads to product changes that solve nothing and confuse the next analysis.
The Workshop works out with your team which of these matter for your stack right now, and what to do first: a 90-minute session with the people who own the decision.
The entry-event definition that anchors the analysis, the activation milestone that surfaces which users retain, the freshness discipline that keeps the curves usable, and the signal-quality checks that separate product moves from tracking moves: those are the decisions that turn an aggregate engagement number into a read on whether the product is actually working over time.
Did this recipe match your situation?Anonymous response. Sign up to leave a longer note tied to your account.
Pick an attribution model, lock it, document why, and reconcile its outputs against incrementality, so the channel report becomes one rule the team can defend rather than a debate per meeting
Watch the conversion events at the resolution that catches a deploy breaking the cart-success fire before reporting starts surprising people, and surface the fix as an operational alert rather than a quarterly investigation
Measure paid channel contribution honestly when tracking is partial, by separating what you observed from what you modeled