There’s no such thing as a bad business metric, Michael Carney told me. But there are bad ways to use them. As a partner at venture capital firm Upfront Ventures, Carney hears investment pitches from hundreds of founders every month. The best pitches seamlessly combine vision and metrics to paint a compelling picture of what a startup’s leaders hope to achieve — and what they’ve achieved already.
Vision and metrics have a lot in common: One tells a story with words, the other with numbers. Both can be thoughtful and straightforward, or unclear and misguided. The key to an effective metric, Carney said, is context.
“You could probably find a metric or set of metrics in any business, even the ones that down the line will be tremendously successful, that raise questions,” he said. “But the more data you have and the more that data is trend-based rather than moment-in-time based, the better story it will tell.”
Sometimes, however, companies would rather tell a great story than the full story. That’s where vanity metrics come in. A vanity metric gives a falsely inflated view of a company’s growth or potential. Often, founders use them inadvertently because they’re drawing on reporting practices that don’t make sense for their company’s model or growth stage.
What Are Vanity Metrics?
“There’s a hundred blog posts that talk about measuring your SaaS business, and every single one of them will talk about revenue churn and lifetime value to customer acquisition cost,” said Guy Turner, a partner at Hyde Park Venture Partners.
“The problem is, when you’re doing 500,000 in annual recurring revenue across customers, and you’ve only had them for six months, everything except the revenue metric is meaningless. A first-time entrepreneur wouldn’t know that, and wouldn’t necessarily be expected to know that.”
Other times, companies use vanity metrics intentionally. This could be entirely benign, like when a marketing department shares a user-count milestone. It could also be shifty, like when a company obscures its margins to appear more profitable.
Vanity metrics may fail to impress savvy investors, but the biggest risk they pose is throwing a business off track. Choosing a “North Star metric,” as Turner calls it, can focus a company’s growth efforts. But if teams pursue metrics-based goals at the expense of broader business health — or the company outgrows the metric it originally chose — companies find themselves out of touch with customer needs.
Below, Carney and Turner share six ways to get more value from your performance metrics.
Instead of a cumulative metric, try a rate of change
Both Carney and Turner named cumulative metrics as top offenders. A cumulative metric shows a business’ performance to-date, instead of changes in performance over time.
“Say you’re a products company and you’ve sold eight million rolls of toilet paper, and you show a chart that goes up and to the right with a cumulative number,” Turner said. “And the response is: OK, cool. Over what time period are you selling more? Are you selling more than you used to sell? You’d be surprised how often we see that with entrepreneurs.”
“You can game those metrics by doing things like paying for downloads.”
Cumulative sales volume is one vanity metric, but cumulative revenue, total users and lifetime downloads can also fall in this category. A strong annual revenue number says nothing about whether your business is becoming healthier or less healthy with time, just as a large number of users or downloads doesn’t mean your applications create value for customers and profit for investors.
Without the context of how your metrics are changing with time, your audience will struggle to draw meaningful conclusions — or write off your success as a fluke.
“You can game those metrics by doing things like paying for downloads,” Carney said. “That’s why app store rankings in a moment in time are not always that interesting. Over time, it can be much more exciting if you have staying power at the top of a list. Otherwise, you could be benefiting from an outlier event that got you a little bit of notoriety.”
Instead of lifetime value to cost of acquisition, try lifetime profitability to cost of acquisition
Lifetime value to cost of acquisition (LTV:CAC) shows how much revenue customers bring in versus how much marketing and sales spending it takes to acquire them.
Companies, Carney said, often leave out information that would make this metric more informative, such as payback time. If it takes a customer five years to spend enough to offset its CAC, the payback time may be too long to keep an unprofitable, fledgling business running healthily, even if LTV:CAC looks strong.
“Payback time is a really important illustrator of business health that usually doesn’t get shared publicly,” Carney said. “But it often doesn’t even get discussed internally when companies are assessing their performance or trying to pitch themselves to investors.”
“Payback time is a really important illustrator of business health that usually doesn’t get shared publicly.”
Another common mistake with LTV:CAC is reporting it too early in a company’s trajectory, Turner said. If a young SaaS company has just a few customers, actual CAC is virtually unknown. The company’s founders likely signed those customers themselves, rather than relying on structured sales and marketing efforts. Similarly, LTV is a mystery because the company doesn’t know its retention rate yet.
To make customer LTV more illuminating, try reporting lifetime profit margin instead of lifetime revenue, Carney said. That shows your audience whether customers will pay back not just the cost of acquiring them, but the costs associated with production, shipping or labor as well.
“That’s the way most investors look at it when trying to truly assess the health of your business on a margin basis. Sometimes [customer lifetime profit margin] is built into LTV, and people just don’t say it out loud. But often, people will either knowingly or unknowingly not build it in, and that tells a very different story,” Carney said.
Lastly, keep in mind that a good LTV:CAC depends largely on your business model. In the SaaS space, for instance, startups can spend larger sums on acquisition and have longer LTV payback times because customers are likely to stay longer and spend more. For a consumer business, CAC should ideally be low, as standout companies find new ways to harness word-of-mouth and viral marketing.
Instead of annual recurring revenue growth, try bookings growth
Annual recurring revenue (ARR) growth shows if your business is bringing in more money year-over-year through its subscriptions or contracts, but it doesn’t reveal whether your sales organization is becoming more effective with time. If your sales team adds 20 new clients each month, for example, your revenue will continue to increase. The team’s skill and market savvy, however, could remain stagnant.
“Bookings growth is a more honest assessment of a company’s growth over time.”
To show off your sales organization’s performance, try reporting bookings growth instead. Regardless of topline revenue, a talented sales team that consistently improves is a strong indicator of a viable business, Carney said. In other words, a useful product or service is great — but can you keep selling more of it?
“Bookings growth is a more honest assessment of a company’s growth over time,” he said. “It’s directly reflective of the new business you’ve added.”
Include revenue sharing information when you report gross merchandise volume
Turner said one of the most common vanity metrics he encounters is gross merchandise volume (GMV), or the total dollar value of a retail company’s sales over a given period. As with other problematic metrics, a lack of context is often to blame.
Many marketplace businesses report the total value of all transactions on the platform instead of reporting revenue. They’ll also use annualized GMV as a run rate, or an indicator of future earnings. However, GMV says little about how much money ends up in company coffers.
Each business has a different fee structure or revenue-sharing ratio. So, a company with a $10 million annualized GMV and a 20 percent revenue share has a very different outlook than a company with the same GMV but a 5 percent revenue share, Turner said.
“That’s not to say there isn’t a relevant application when GMV can be a useful metric,” he added. “It’s that it’s often used as a substitute for revenue, which I wouldn’t say is misleading, but it’s often unhelpful.”
Consider which metrics make sense for your growth stage
Some metrics are useful at certain growth stages and insignificant at others.
Turner grouped performance metrics into three categories: leading indicators of success, real-time indicators and lagging indicators. For a new startup, reporting a real-time indicator like revenue or a lagging indicator like churn doesn’t make sense.
Instead, new companies should zero in on the metrics that suggest their products or services will continue to be useful for customers. For a SaaS business, that may be the numbers of pilots and, more importantly, the pilot customers’ activity. If customers log in frequently and use a variety of features, that signals strong revenue potential down the line. And while these leading indicators may feel less concrete, they’re more informative than made-up retention or churn metrics.
“The period over which that metric is being calculated has a significant impact on what would be viewed as exceptional and what would be viewed as not exceptional or, frankly, irrelevant.”
“You only find out about churn after the fact,” Turner said. “If you want to know it beforehand, look at whether your customers are actually using the software. If they’re not using it now, later they will churn.”
Be wary of assigning too much significance to user activity metrics too early, though, Carney cautioned. An app that’s been live for a week, for instance, will likely have an inflated number of active users.
“The period over which that metric is being calculated has a significant impact on what would be viewed as exceptional and what would be viewed as not exceptional or, frankly, irrelevant,” he said.
Define your metrics transparently
The easiest way to avoid vanity metrics is to clearly define what you’re measuring.
Both Carney and Turner named active users as a metric that frequently lacks definition. For one company, an active user could be someone who received a push notification. For another, it could be someone who completed an in-app economic transaction. Because of that, some active user metrics have implications for revenue, while others don’t.
Adding that context, Carney said, turns active users from an empty number to a valuable insight.
“It’s interesting to know how many users there are on an absolute basis, but it’s much more interesting to know what they’re doing with your product and how important your product is to their day-to-day lives,” he said. “From that, you can infer how long they’re likely to stick around and how much they might be willing to spend to participate in your product or service.”
“It can be a case of someone being nefarious, but I think more often than not it’s people not understanding the definitions or not being precise.”
ARR is another frequently misused metric. The term is so common, Carney said, that companies without recurring revenue models will use it to report their earnings. An e-commerce business without contracts or subscriptions, for example, should not report ARR.
“People will say, ‘We have this much in ARR.’ But what they’re really saying is, ‘Last month we generated this much revenue. If you multiply that by 12, it’s our current run rate.’” he said. “But that’s mostly irrelevant and misleading. It can be a case of someone being nefarious, but I think more often than not it’s people not understanding the definitions or not being precise.”
Finally, use caution when employing metrics that aren’t standard. Explain clearly how the metric has been adjusted and why that’s an appropriate fit for your business.
Take adjusted earnings before interest, taxes, depreciation and amortization (EBITDA). Some companies may report an adjusted EBITDA because it makes more sense for their business model than the standard metric. If they’re transparent, it’s probably no issue. Other companies, however, may use adjusted EBITDA to hide alarming cash burn or other cost structure problems.
“The most important thing is knowing that the receiver of the metric and the producer of the metric are speaking the same language,” Carney said. “When you know the inherent assumptions and definitions, you can bake those into your assessment of whether that metric tells a good story.”
