Billy Beane, general manager of the Oakland A’s in 2002, lost his three best baseball players to free agency and had little cash to spend on other superstars. Instead, he focused on using a specific set of metrics — slugging percentage and on-base percentage — to build a team of undervalued, unheralded players, much to the bafflement of the baseball world.
The result: Beane led the A’s to the fifth-longest winning streak in baseball history and three American League West titles in five years.
For software development teams, metrics can similarly be used not only to understand how productive and efficient a team is at a variety of tasks, but also to predict future changes in that efficiency and address any roadblocks along the way. Rather than early-stage development, DevOps metrics usually focus on deployment, operations and support. To that end, the metrics themselves are frequently broken down into three specific categories: people (such as response time); process (lead time); and technology (uptime).
While some of the more common measures are known to many DevOps teams, other lesser-known metrics can also have a major influence. Much like Billy Beane focused on specific metrics to unlock new understandings of baseball productivity, we sought to find unique data points that do the same for software engineers. We sat down with three engineering leaders to discuss which software development metrics they use to measure the productivity of their engineering teams.
Best Software Development Metrics
- Deployment frequency
- Sprint velocity
- Uptime
- Mean time to recovery
- Lead time
- Change failure rate
- Points completed per person and per day
- Predictability rate
- Team Happiness
Fivetran
Teena Mathew
ENGINEERING MANAGER
What metrics do you use to measure the productivity of your engineering team, and why?
The teams I work with at Fivetran track sprint velocity, incident incoming rate and code debt. Sprint velocity tracking enables teams, specifically engineering, product and leads, to look at data from previous iterations and commit to what is doable.
Regarding incident incoming rate, my teams prioritize customer issues based on a prioritization matrix. Keeping track of the number of incoming issues logged by customer support, sales and internal teams helps us stay agile and re-prioritize when necessary.
We track code debt to ensure code stays maintainable. It also helps in keeping correction costs at a minimum.
By refining and standardizing story pointing, the team understands and discusses the complexity of a task without basing it on an individual’s knowledge.”
What are some examples of insights you’ve derived from these metrics, and what actions came out of them?
Initially, I noticed many variations in sprint velocity. For instance, on a team of five, there were sprints with 100 story points and some with 45 story points. This variation compelled us to take action in a couple of ways.
First, we realized the identified story pointing was not consistent, so we ran through training for story pointing with our teams using examples from our future tasks.
We also needed to normalize sprint velocity to load balance. We realized the difference in the number of story points completed needed to be plus or minus five points to get to a comfortable commitment rate for our teams.
What impact has tracking these metrics (and acting upon the data) had on your team’s productivity?
By refining and standardizing story pointing, the team understands and discusses the complexity of a task without basing it on an individual's knowledge. This has helped improve the velocity of the team since we now lean on each other for task completion.
Keeping track of incoming incident rates helps us improve and feature velocity. We plan future quarters based on the type of incoming incidents and engineering priorities that include feature proposals based on the incident types. This has increased our feature delivery velocity over time while reducing the incoming incident rate.
Bullhorn
Ben Levine
VP OF SOFTWARE ENGINEERING
What metrics do you use to measure the productivity of your engineering team, and why?
At Bullhorn, we leverage a variety of metrics that are intended not just for leadership consumption and review but for the whole team to use and digest. We believe the team is best positioned to see changes in data and act on it quickly, so everything we review at the leadership level is open and visible. Metrics that we leverage to access productivity include velocity (measure of story points completed per sprint), points completed per person, points completed per day (this includes developers and testers), hotfix deliveries per release (hotfix is any ticket delivered outside of a planned release candidate), minor releases per cycle and size of releases.
Additionally, our metrics include the percentage of planned versus unplanned tickets (planned tickets are those committed to by the team at sprint planning), percentage of tickets groomed at the start of the sprint, unit test coverage per repo and automation test coverage.
We leverage this set of data points because it allows the teams and their leaders to gain a holistic view of the health of the team and each project. It also gives us the ability to deliver a consistent product to our customers while reducing the benefits of trying to game the system by seeking to drive one measure in an unnatural manner. And lastly, it allows each team to see early indications of trouble and pivot quickly.
The metrics help the teams constantly improve.”
What’s an example of insights you’ve derived from these metrics, and what actions came out of them?
In leveraging the review of hotfixes and deployments per cycle, we found that the level of hotfix delivery and associated churn was being amplified by the smaller but rapid nature of hotfix vehicles being delivered. Since each vehicle created a level of overhead and need in deployment, by narrowing down the number of vehicles to set days of the week and packaging a couple of items in each vehicle, we were able to reduce the churn in the teams and provide for more consistency in sprint delivery.
In addition, in looking at both velocity and points per person per day, we can see how changes in team capacity and alignment have an impact on the team’s delivery capacity. This helps us to understand the impact of making resource shifts across the teams and roadmap, and allows us to be much more deliberate and strategic in making team changes to minimize the disruption that the changes can cause. It also helps us to ensure we are still supporting changing customer and business needs.
What impact has tracking these metrics and acting upon the data had on your team’s productivity?
Tracking these metrics and, in particular, making them a part of the conversation that the teams have in retrospective and other forums, has had a positive impact on the teams’ productivity.
The metrics help the teams constantly improve. It allows for direct and timely feedback on the impact of changes that the team makes. It also allows them to understand what’s working and what isn’t. That information, coupled with a wider review, enables our teams to share ideas about how exactly to improve productivity.
Through the course of the pandemic, we saw our teams continue to improve in their delivery numbers across the board despite the chaotic nature of the shifts in work environment and world state occurring around them.
Overhaul
Jason Vertrees
SVP OF TECHNOLOGY
Jason Vertrees is senior vice president of technology at Overhaul, a supply chain integrity solutions company. While his engineering team uses a variety of metrics to measure productivity, he has zeroed in on two useful measures in particular that have unlocked greater models of velocity and prediction at work.
What metrics do you use to measure the productivity of your engineering team, and why?
There are the standard measures many companies focus on, including us, like uptime, mean time to recovery, lead time, change failure rate and deployment frequency. Most know about those, so I’ll talk about something else.
Two that I’ve found useful outside the above are “team happiness” and “predictability rate.”
What actions came out of these insights?
“Team happiness,” while subjective, can be tossed onto a Likert-type scale giving you the opportunity to quantify it. In addition, happiness is a leading (not lagging) indicator: if your happiness goes down, your upcoming velocity is going to decrease as a result.
Predictability helps us measure the percentage of large projects delivered on time. It helps us hone our estimations, which are critical for external teams’ planning.
Happiness is a leading (not lagging) indicator: if your happiness goes down, your upcoming velocity is going to decrease as a result.”
What impact has tracking these metrics (and acting upon the data) had on your team’s productivity?
Intentional focus on happiness helped us uncover the sources of lost productivity.
I used this on one project and at one point we noticed a dip in the team’s happiness. Once the decreased scores were spotted, that spawned a line of questioning in the retrospective where two non-technical reasons surfaced: lack of clear product preparation/requirements and developers’ sense that management wasn’t actively listening. After knowing the cause, we were able to address the issues, restoring that team’s great output, happiness and trust in management.