The Growth Process

In the previous step you split an unwieldy goal into prioritized components with individually measurable metrics. In this step we’ll do all of the research, both qualitative and quantitative, and pre-experiment work that’s needed to understand how we’ll actually move this metric. This research gives us absolute confidence that we’re targeting genuine opportunities.

Experiments shouldn’t be guesswork. When an experiment fails, it can be for two reasons:

1. The design wasn’t effective at changing the users behavior
2. The design didn’t target a real need of the users

This research aims to ensure that we’re targeting genuine opportunities; giving us the confidence that failures are a result of ineffective design and ensuring we’ll achieve success more often.

1. Research preparation

Create a research plan that clearly explains your objectives. Ultimately you’ll aim to improve or add features that facilitate an action that you want users to perform more frequently. At a minimum you’ll need to know:

• What does the user want to accomplish. What are their goals or needs?
• How do users go about doing it? How does it work today, versus what are their expectations about how it should work?
• What prevents users from doing that task today?
• Who should be doing this task? Is it all users, or a segment (e.g. only admins).
• Are they using work-arounds or alternative solutions?

2. Perform qualitative research

There are many methods that you can use to perform qualitative research. Some examples:

• Competitor analysis - Review competitors in the same and peripheral industries. In most cases they will have solutions to the same problems you aim to solve. Often the solutions will not be directly applicable, but we can use the info to understand best practices / patterns, inform new experiment ideas, etc.
• Customer interviews, contextual inquires, diary studies, etc. - This is our primary method for understanding what our customers want. Conduct enough research to establish clear patterns, and then summarize those patterns into a model of user behaviour. Often there’s a mismatch with what customers expect/want, and what we deliver. Good experiment ideas eliminate this gap.
• Heuristic review - Do an end-to-end exploration of our products from the perspective of a persona that you are focusing on. Document what the current experience achieves and where are there obvious problems.

3. Create and measure a funnel

In most cases, users will need to complete a series of actions in order to contribute to your metric. Your funnel should list all of these steps and the rate that users complete them.

This funnel is important for several reasons:

• The funnel identifies which features work well, and which don’t. If particular parts of your funnel have low completion rates, you can focus initial efforts on solving those problems. In most cases, low points in your funnel can be mapped to pain points in your qualitative findings.
• This funnel empowers you to work incrementally. By using a funnel we know that improving individual steps in the funnel may not move our end metric, but a combination of smaller improvements across the funnel will.

4. Perform quantitative analysis

We aim to understand what behaviors make a user successful. Define two cohorts of users: those that are successful (i.e. they contribute positively to your metric) and those that are unsuccessful (they contribute negatively to your metric). With these cohorts defined, the analyst can use product analytics event (or you may need to implement/improve events) to create decisions trees, dendrograms, context matrixes, etc. that describe correlations between events and users that are (un)successful.

By combining these correlations with the “why” (qualitative) you’ll understand how to design effective experiments.

5. Define experimental baseline

You know which measure you aim to move, but haven’t yet determined if it’s experimentally viable. For a measure to be appropriate for experimentation it must be:

• Rapid: We must be able to measure our metrics within 1 week of running the experiments in order to maintain a fast experiment cadence and to quickly iterate on work. In some cases you may be willing to run experiments over longer periods, but these experiments should be ‘big bets’.
• Valuable: The metric must directly increase your end goal, or otherwise: 1) Be a component of a metric funnel, or 2) there is a high level of confidence that moving a correlated metric will result in a corresponding MAU or funnel change.