The Rule of 70%

This article was originally published by Ameet on Medium.com.

{{divider}}

As product managers, leaders, and entrepreneurs, we are often confronted with situations where we must make a decision, but where there is uncertainty and incomplete information.

A good example I faced personally was back in 2012, when I was a product manager on the Twitter Ads team and was working on a new Brand Survey ad measurement tool. The tool required us to build a completely new survey format that would appear in users’ Home Timeline — something that had never been done before. We faced a lot of unknowns about engagement rate (positive and negative), impact on user retention, survey completion, and potential backlash from the Twitter user community.

We had gotten some feedback later in our development that challenged our assumptions around the design, and could potentially impact our success metrics. Our team had to decide whether to ship the current design, or increase our confidence through additional design prototyping and user testing before shipping. To help us make a decision quickly, we turned to the Rule of 70%.

Before I get into what the Rule of 70% is, let’s start with why this is important.

We’re trying new things in our product every day. Some of our experiments will succeed, others will fail. But in every case, there’s uncertainty about the outcome. It’s our job to get customer learnings quickly and then iterate our product as fast as possible based on these learnings.

I often remember this quote by Benjamin Franklin: “Lost time is never found again.” This is especially true in a competitive market, where there are multiple players vying to become the leader.

{{quote}} The winner will be the one who can identify customer problems and solve them the fastest, the one who can adapt most quickly to evolving customer needs.

One of the things that slows down product teams is taking too long to make a decision. This was a risk we faced back in my example from Twitter. It’s probably a risk that every product team often faces. This is where the Rule of 70% comes into play.

Basically, the Rule of 70% is that we should make a decision when we’re 70% confident. This is based on the following insights and assumptions:

• There is a fundamental trade-off between accuracy (how correct is our decision) and speed (how quickly do we make the decision)
• At a certain point we see diminishing returns as we invest more to increase confidence
• Due to these diminishing returns, the cost of losing speed (and time to market) begins to outweigh the benefit of the additional certainty
• If you continue to invest time and resources to increase your confidence beyond 70%, you’re likely making the wrong trade-off between accuracy and speed

I often see product teams make the mistake of over-investing in increasing their confidence — through seeking feedback and approval from stakeholders, considering and analyzing more risks, doing more research or design prototype testing — beyond the 70% threshold. You can use the Rule of 70% to avoid this trap.

Ask yourself and your team the following question: How confident are we in this decision?

If you’re already at 70%, make the decision and go forward. If not, then only invest the minimal amount of time and resources to get to 70%, and no more. Ruthlessly scope down any additional analysis, feedback collection, prototype testing, etc. so that you don’t end up on the wrong side of the accuracy/speed trade-off.

There are two questions I frequently get when I share the Rule of 70%:

The first question is, “How can you get to 70% quickly?”

A few things come to mind. First, you can look at what competitors and analogous solutions do. If it works for them, this should boost our confidence that some version of it could work for us. Second, you can consult with experts to get quick input. Chances are there is someone in your network who has faced the same or similar problem in the past — get input from them about how they approached it, and what they’ve learned. Third, you can do some quick hallway research.

Even by testing your solution ideas or designs with a few trusted colleagues, you will probably identify major risks or usability issues and increase your confidence in your decision. All of the above can be done relatively quickly, in a matter of hours, or perhaps a day or two. I always urge teams to timebox how much time they allocate towards getting to 70% confidence.

The second question: “Is 70% a universal threshold?”

No, it depends on the stakes. If your decision is low stakes, you can relax the 70% further down to 40–50% or even lower. One of the nice things about opportunistic testing — continuously running low-cost experiments — is that the stakes for the decision are low. Since it’s a small change and a small resource investment, if your decision is wrong and the experiment fails, it’s fine.

You don’t need to invest a lot of time to create a well-developed hypothesis (although every experiment should have some plausible hypothesis for the customer or business problem). You can try something small and see if it works. This is why I advocate for teams to always have some budget for opportunistic testing as part of their product portfolio.

Conversely, if the decision is high stakes, then you may need to have a higher threshold. However, most of the decisions we make in product development are reversible decisions — so the likelihood of confronting a high-stakes decision that would require more than 70% confidence is low.

Going back to the original situation in 2012: when we were faced with this launch decision, I got together with our EM, designer, and a few engineers on the team. I took the pulse of the team about how confident we were in the decision to ship. The majority of us were already at 70% confidence.

The ones that weren’t raised some good points, and we decided that we would focus on these risks after the initial launch if the performance didn’t meet our expectations. Just using the 70% rule as a conversation starter with my team helped to create alignment around the decision, as well as surface some valuable inputs for our next iteration.

We ultimately shipped the first version of the product, and it performed well enough in the field that we didn’t need to further iterate the design at all. This was an important lesson for me — we could have spent more time iterating the design, testing it with users, and increasing our confidence. Ultimately, the true test of the product was how it performed with real users in the field. It was better to get the actual learnings from real users than it would have been to continue iterating the design pre-launch.

I urge all of you to consider using the Rule of 70% to help you make decisions quickly with your team. Since we’re innovating, we are always trying new things and there are always unknowns and uncertainties.

Our job is to get customer and business learnings quickly, and rapidly iterate our products based on those learnings. In a competitive market, lost time has real cost. When faced with a difficult decision, we can use the Rule of 70% by asking ourselves, “How confident are we?” If we’re at 70% confidence, make the decision and go. If not, timebox and scope down as much as possible our investment to get to 70%. We want to end up on the right side of the accuracy / speed trade-off.