I worked at a company with one of the best-funded and advanced analytics organizations in the industry. My product pod had a full-time analytics person, whose whole job was to help me validate ideas through A/B testing, regression analyses and deep-dive investigations. The company boasted a culture of data-driven decision making toward our North Star metric. I felt confident making decisions, knowing that my roadmap and backlog were prioritized using real, hard data.
Until they weren’t.
I soon took on a relatively new app team with one big problem: they had no analytics tagging anywhere in the product. They didn’t have an analytics person or access to the data tools I relied on. We were flying blind, and I had about a month to onboard and develop the following year’s roadmap (the timing was terrible).
3 Ways to Make Decisions Without Data
- Find prioritization networks that resonate with your style of management.
- Use proxies through opportunity sizing.
- Talk to your users and understand what they need.
I struggled, hard. Without any app tracking data or basic customer information, I didn’t know who my customer even was, much less how to advocate for them. I found myself paralyzed by the uncertainty and fear of making the wrong decision, totally unable to find a way to prioritize all of my ideas.
While a bit extreme, my experience on the app team is not unusual in product management. I’ve learned that it’s incredibly rare to have access to all of the data you’ll need to make good decisions. But the days keep rolling, ideas keep popping up and sprint planning comes every two weeks. You have to be able to prioritize through uncertainty or else competing interests will do it for you.
This is why, in every product manager interview, one question separates the novices from the pros: How do you make decisions without data?
Data-Driven Decision Making Is Important, Right?
Data-driven decision making is no longer a skill that differentiates a product manager. It’s a junior-level, baseline skill, and there’s no shortage of product analytics tools at our disposal. If you work in e-commerce and your goal is to optimize a single page or button, then you’ll likely have a clear North Star metric and a decent amount of data to work with.
What will you do when your mandate is to bring a brand new product to market? Or, to solve a problem that will take a year of engineering work before it can launch? Or in my case, how do you build a year-long roadmap in a short period of time when you have no hard data to work with? Product leaders differentiate themselves from the novices by using strategy skills to make decisions with limited data. The good news is that this skill can be learned through practice.
How to Make Decisions Without Data
Many PMs call this skill “product sense,” or an innate ability to be able to prioritize based on your past experiences and instincts. A more skilled approach would be to leverage the information and data that you do have to make good decisions.
To start developing this product sense, find a few prioritization frameworks that resonate with your style of product management. There’s no shortage of them out there. Practice prioritizing your backlog using different frameworks and see which trends emerge. After a few years of doing this, I developed my own two frameworks, The 3 Principles of Product Success and The 3 Most Important Questions in Product Management, which I still use to triage incoming work.
Another way to prioritize without data is to use proxies through opportunity sizing. Proxies are products similar to yours from which you can reasonably borrow data and make assumptions. For example, when I was working on promoting a B2B program on our website, I was able to borrow the clickthrough and success rates from our loyalty team, which used similar methods to promote the loyalty program. Opportunity sizing uses the limited data that you do have, along with thoughtful predictions, to help you prioritize your backlog.
Finally, nothing beats talking to your users and understanding what they need. According to a study from 280 Group, “Product managers who spend 30 percent or more of their time engaging externally lead to improved performance for both the product team and the organization.”
However, this study also shows that only 10.9 percent of product managers are able to engage at least 30 percent of their time with their users. Knowing what your users need and why they chose your product is crucial information to leverage when you don’t have hard data guiding you forward.
The Takeaway
In the end, I did not create a great year-ahead roadmap for the app. Only about 30 percent of my projects were approved the first time around. I knew this was a likely possibility, and it stung a little.
But I don’t feel like I failed, either. With very limited time and data available to me, I used the frameworks and tools that I had to make the best decisions that I could at the time. The good news was that 30 percent of our projects were approved, which gave the engineering team about three months of runway to work on high-value, high-impact work. Meanwhile, I focused on fixing our analytics tagging, revising my assumptions and frameworks, and interviewing our customers so I could better prioritize the work ahead of us.
The most important part of the product manager role is being able to ruthlessly prioritize the highest-value, highest-impact work so you can deliver the best experience to your customers. While many junior PMs believe that you can brute-force your way through a backlog using exclusively “data-driven'' methods of prioritization, the real-world challenges of product management require softer skills and product sense to make the best use of your team’s time.
There are no perfect decisions in product management, so you need to be able to make the best decision you can with the information and resources that you have.
