Do You Actually Need to Build That Product?

In product management, we’re always under the gun to innovate. But according to Ron Kohavi, a former executive at AirBnb and Microsoft, “60 to 90 percent of ideas do not improve the metrics they were intended to improve.”

Even if you believe that number isn’t accurate for some reason, the truth is probably still grim. Let’s cut Kohavi’s estimate in half and say that 30 to 45 percent of ideas add no value or, even worse, have negative value. Does your team’s decision-making account for these percentages? 

Over the last 15 years, I’ve advised dozens of teams to help them learn how to build products that are valuable to customers. Whenever I tell my clients about this ratio, the team members’ eyes generally get wide. Suddenly, they realize that removal is also a part of the product development process.

Product development and the product manager’s job, which we can describe as helping teams make better decisions consistently, isn’t linear. As much as investors may want our work to go up and to the right, product development is full of peaks and valleys.   

Inside those peaks and valleys lies complexity that can be hard to see. The complexity includes the work that goes into keeping an idea alive, which can be summed up by the saying, “There is no such thing as a free lunch.” 

Think about what happens after launch day and the questions that arise the minute a product lands in production: 

• How will we handle the maintenance of the tools?
• How will we have to build around that tool in the future?
• What new functionality will affect other parts of the product?

If you aren’t thinking about these things, I can guarantee there will be a surprise in your future. Nothing we build is disconnected from what we’ve built before. This reality is the foundation of tech debt: Every time you deliver something, you accrue debt that needs consideration.

Sounds overwhelming? What we have above is but a fraction of the things you’ll really encounter on the job. These items, plus the other twenty things we could list here, compound when you think about how a release affects the larger business and customers. These problems can become legion when thinking about how many releases are happening at once in our continuous integration, continuous delivery (CI/CD) world. 

So, on the one hand, we have the reality that most ideas don’t work. On the other, we have a mountain of debt that we face in the product world that can overwhelm us if we deliver too much.

Sounds like a lot? Well, no worries. We have a tool in our kit that helps us invalidate the ideas that aren’t moving us in the right direction (negative outcome) or are not worth the debt (neutral outcome) and move on to the things worth stressing about. 

That tool is called a discovery process. A healthy team looks to discovery to find ways to invalidate an idea, which it shifts into a falsifiable hypothesis to understand its worth. 

To help you make the most of this tool, let’s look at some questions you can ask during discovery that are helpful for invalidation, the common mistakes teams make that turn discovery into a validation engine, and what you can do to eliminate those mistakes moving forward.

The World of ProductWhat Does a Product Manager Actually Do?

Let’s Talk About Discovery

The biggest mistake that teams make during discovery is framing the discussion around validation. By starting with that concept, you open the door to confirmation bias at the start of the process. If you are thinking about how you’ll validate an idea, chances are you’ll start to look for ways to make it so. This approach can subtly kill a discovery process.

Instead, we should frame the process around the opposite idea — invalidation. When we are doing our discovery process, we should look for things that aren’t going to work so we can move on from them as fast as possible. Let’s start with that frame and add some fire to it.  

How do you begin to frame an invalidation process? Well, one of the better ways of defining discovery comes from Marty Cagan’s Inspired, where he frames the process around risk rather than validation. This can take multiple forms:

So, before we move on to anything else, stop here. Ask yourself if your discovery process is seeking validation or invalidation. This question can serve as a quick heuristic: “Have any of my discovery efforts failed recently?” Remember, 60 to 90 percent of ideas provide neutral to negative value; it shouldn’t take long to hit invalidation. 

The mindset shift is important because you can transform a process that often ends up siloed into a cross-functional one. 

Your research should be cross-functional for many reasons, but the most important one is this approach means that you’ll figure out if something isn’t worth following up on much earlier in the process. The risk-based frame expands the conversation, making the session less focused on whether an idea is “bad” by giving insight into whether the solution is worth the effort. 

Remember, the work on a product only begins when you launch. If risks create a situation where the debt incurred won’t be worth the focus, you have an opportunity to get rid of something that doesn’t move the needle. The questions above, asked of the right people, create opportunities to see this before launch.

Invalidate on Convergence

Let’s make this real by visiting BobCo, a fintech company that’s working on a brand new onboarding functionality. The product manager on the initiative, Rachel, finds herself in the middle of a discovery session. 

The good news is that she’s decided to switch up her approach. In the past, she’s made the mistake of building functionality that hasn’t found a ton of traction. Her last few releases are around the 15 percent (plus or minus 5 percent) usage mark. For most places, that doesn’t move the needle, and now her team is managing these features that aren’t making much headway in the marketplace. She’s determined to avoid that problem this time around and build something that really delivers for the company.

So, she begins by looking at the list of problems that she came up with. Instead of her old routine in which she kicked the list around with her design peer and then went right into talking to customers, she decided to ask a few questions to a few different stakeholders.

• Value risk: Will the customer buy or choose to use this solution? (Product Marketing/Presales) 
• Usability risk: Will the customer understand how to use this solution? (Design) 
• Execution risk: Do we have the capacity to build this solution? (Engineering) 
• Viability risk: Does this solution work for our business? (Customer Success, Support) 

Rachel used a very simple impact versus effort questionnaire that asked stakeholders to rate the potential impact and how much effort they think the solution will take on a scale of one to 10. She framed the questionnaire around those questions above. She wasn’t looking for in-depth explanations, just some signal from various departments on what was going to be useful. 

She didn’t call large-scale meetings to do this research, either, but just asked her questions via Slack. These check-ins with the various stakeholders took about an hour combined, and the list of problems that made sense to worry about shrunk significantly because she was able to score them based on the questionnaire. She then decided to tackle the problem that had the highest impact with the lowest amount of effort first. 

This method had an extra beneficial effect as well. Suddenly, she had more advocates around the business for her research since they understood it better. People discussed her work in a bunch of different rooms, and her updates on the discovery had eyes on them. 

Note that you want a minimum score to begin discovery. You don’t want to work on things that have low impact just because you need to “stay busy.” If you have the time, go back and analyze impact versus effort for some of your previous initiatives. This retrospective will help your future efforts because you’ll want to beat the average on anything you start now. 

More in ProductSolve Any Product Problem With the Design Thinking Framework

Ideas Are Cheap — Commitment Is Expensive 

Here are some questions to ask of yourself throughout discovery:

1. Are you framing discovery as invalidation instead of validation? Have you disposed of an idea lately?
2. Are you thinking about discovery as a cross-functional exercise? Who is involved in the process? Is there a way you can maximize signal over noise?
3. Are you thinking about tech debt risk?
4. Are you thinking about a minimal score. Things have to be worth tech debt: Is this one? 

When we start with invalidation as the goal instead of validation, we are more prepared to get rid of the things that don’t matter. Every step we take as product managers is a step towards helping our teams make better decisions, and driving simplicity through invalidation is a big part of that.

Remember, less is more.