How to roll out and use engineering metrics successfully
“Help, my boss is asking me to come up with metrics!”, or “I want to use metrics for my team(s), what do I do?” These are situations that many of us have dealt with, and I’m here to help. The good news is: The steps to make metrics work for you and your team(s) are the same in all cases. In our joint 35+ years of experience, my co-author Maggie Litton and I found that the following steps are critical for making sure metrics are actually useful, no matter what level you’re at.
Metrics have been a topic of hot debate in tech for a really long time, and in an industry where being on top of the latest trend is seen as a virtue and pattern-matching is so common, it’s easy to feel pressure to follow the latest trends. But: What works for one organization will likely not work for yours without adjustments. Take this example: I currently work with two similarly sized companies that are both very successful in their respective, adjacent fields. One has been in business for 25 years and is just introducing unit tests; the other company is fine-tuning its continuous deployment setup.
So how do you successfully introduce metrics to your team(s) or department?
Step 1: Understand the problem you’re trying to solve
Whenever I speak with leaders who tell me they want to introduce certain metrics, I like to dig into what they’re trying to achieve. With no exception, it becomes clear very quickly that their actual concern goes far beyond a specific metric, and is instead something like:
We’ve hired a lot of people, why are we not getting as much done as I’d expect?
Why is project x taking so long?
Why did we just have another incident in $area?
What are we doing to fix $problem?
I’m getting questions from my fellow executives or the board that I don’t have answers to.
I feel like my team(s) don’t have a good grasp on a certain area/topic.
Here’s what this can look like in practice:
Case study: “Story Points”
Recently, I worked with an engineering executive team that wanted to start measuring story points across their teams. As I learned more about their goals, it became clear that they were concerned about a project that was supposed to be released months ago, but kept getting delayed.
Eventually, they decided to work with their teams directly and help them scope out the project more clearly, and make changes to their planning, communication, and stakeholder management. Story points wouldn’t have helped here (story points are also not a metric, but more on this below.
No matter who asks you for metrics, your first step should be to understand the problem that needs to be solved. Get curious, and, for as long as you can, focus the conversation on really understanding the problem:
What’s the problem they’re trying to solve? What outcome(s) are they trying to achieve?
Are they trying to learn?
Are they trying to measure an effect or progress? Maybe there’s been a change in your architecture and they want to know if it reduced user-experienced latency as much as they hoped it would.
Are they trying to gain support for a project? Or maybe you want to highlight the impact of toil work on engineering opportunity costs?
Is their goal to satisfy a directive from their boss? Or give them better visibility into the team(s) you run?
Do they want to improve in a specific area like reliability, quality, effort, cost-effectiveness, efficiency,...?
Who needs the solution to this problem? Who’s the audience for these metrics? How will the metrics be used? By whom?
What information are they missing? What would be helpful/persuasive for them?
How quickly do they need this information?
How often will the data need to be updated?
How high a priority is this? [If needed] What do they want you to deprioritize to get this done? What time and resources can you use for this? This is important if obtaining the data requires automation or telemetry investments.
Ask them to explain the connections they see between your team and the organizational metrics they’re collecting. This is especially important if they rely heavily on proxy or composite metrics that can mask how your team contributes to those numbers.
How can you involve the teams in coming up with metrics?
How do they want to handle feedback from team members about this?
What makes them believe these specific proposed metrics will help solve those problems?
Step 2: Determine if metrics are really (part of) the solution
Once you understand the problem well, you can determine if metrics are a useful solution. In my experience, many challenges require a dual approach:
Introduce metrics combined with context. I’ve found that it’s rare that a single metric can answer a complex organizational question. And metrics aren’t much without context: Why is this the number we’re seeing? Is it good, bad, neutral, or do we not know?
Address cultural, structural, and procedural issues, such as poor communication and stakeholder management, dysfunctional team structures, weak delivery practices like low predictability of delivery, and poor expectation management.
Case Study: “Story Points”
The leadership team ultimately took the following dual approach:
Introducing metrics: The teams started using story points as an internal tool for improving their planning practices. They also realized that lack of focus was an issue and introduced a WIP (work in progress) limit.
Addressing issues: The teams made a lot of changes to their delivery processes, such as working more closely with stakeholders, spending more time on planning, and breaking work into smaller chunks.
Step 3: How to pick metrics to set your mission up for success
Let your employees decide what they want to track for themselves: Let folks who report to you come up with their own metrics, even if you don’t fully buy into them. In the last article, I talked about the important difference between metrics for and metrics about teams. Let your teams pick the metrics that are for them to use - after all, those metrics will be most valuable when they help your employees actually learn and improve.
Use metrics for which data is easy to gather. Don’t make people jump through unnecessary hoops for things they don’t care about. There’s a balance between having 15 mandatory JIRA fields to get as much consistent data as possible, vs requiring none and getting no useful data. There can be cases when you want to use metrics for which data is harder to gather; keep those to a minimum.
Don’t use metrics that aren’t actually metrics. Many engineering teams use numbers that may look like metrics but aren’t actually metrics. Common examples include velocity, lines of code, and similar output measures. Stay away from those.
Don’t be afraid to keep multiple sets of metrics: Some for your own managers, some for folks who report up to you, some for specific teams. Not all metrics will be relevant to everyone at all times, and that’s okay.
Be willing to accept a metric that you find sub-optimal, especially if it buys you relationship capital or if you can balance it with other metrics. In the first article in this series, I talk about this more using the example of the cumulative flow chart metric, which often isn’t super useful for teams, but very valuable in aggregate. More examples:
Your leadership may be interested in measuring how quickly new hires become productive. You might be asked to use a metric like time to first pull request (PR). Like most metrics, this one has limited value when used in isolation. A new hire’s first PR might be a trivial exercise designed to give them a sense of how the build pipeline works. Your onboarding process might be designed to ensure all new hires put up a small PR within their first two weeks. This doesn’t mean they’re capable of working independently in a codebase. Rather than strongly objecting to a metric your leadership is fond of, you could try adding an additional metric like self-reported feelings of productivity as part of an onboarding survey, or time-to-first-PR and time to nth PR, e.g. time to 10th and 30th PR.
I once worked with a colleague who was eager to measure cyclomatic complexity in a codebase. I didn’t think this metric would translate well for leaders outside of engineering, but my colleague was very excited about it and made it a hobby project. I decided that including this metric in our charts wasn’t going to hurt anything as long as we also had another metric that was less esoteric. As it turned out, my colleague never finished their hobby project–problem solved, with no hard feelings!
Only gather as many metrics as you’re actually able to use. Resist the temptation to gather a ton of metrics, just because you can. Metrics isn’t a pokémon-style “gotta catch ‘em all”-exercise. I often see leaders trying to measure everything they can. Not only does this become very expensive very quickly, in terms of time, tooling cost, manual labor for data entry, data analysis, etc. It also very quickly defeats the purpose: Giving you insight, spotting patterns, making better decisions, and improving.
This post includes an extensive step-by-step guides to help you identify the metrics that are best for your team(s) or organization: How to choose and use metrics.
Step 3.5: When you have to ask your direct reports to come up with metrics
You may be a middle manager or technical leader who was asked by your boss to come up with metrics. In this case, you’ll next need to involve the line managers or technical staff who report to you. Here’s how to do this well:
Explain the problem you’re trying to solve. Share context! You can use the questions above as a checklist for what you should cover, from the problem you’re trying to solve, to the time and resources they may use for this.
Set expectations for…
Standardization and experimentation. Your line managers need to know when everyone must measure things in the same way. And they will be more engaged if they also have room for some autonomy and customization
Numbers that you and your line managers can easily explain off the top of your head. Steer people away from overly complicated concepts or statistics that will be hard to translate to executives or other functional areas. (Like that cyclomatic complexity example below!)
Metrics that convey the connection between engineering work and business value. It’s fine if teams also want to track metrics that aren’t applicable beyond team boundaries. But that probably isn’t sufficient for middle managers’ and executives’ needs. Take the time to talk with them about how teams contribute to business value; this part is often clearer to a middle manager who has a broader context, and can be more difficult for a line manager.
Do you have a target in mind? Are you looking to measure trends or absolute values?
Line managers and higher-level engineers like staff and principal engineers or tech leads need to be able to use metrics. It’s an important skill to round out their leadership toolkit.
Step 4: Roll metrics out thoughtfully and transparently
Show and earn trust with transparency: Even when you as a leader introduce metrics with the best intentions, beware that many employees have seen metrics weaponized against teams and individuals in the past. They’ll likely have concerns: Leaders are saying one thing now, but if leadership or direction changes, is this going to still remain the case?
How you roll out and use metrics matters. When you share with your team that you plan to introduce metrics, share:
Your intentions: What are you trying to achieve? This may seem obvious to you; assume that not everyone understands the business or specific metrics well yet.
Context: Where is this coming from? What goals are you pursuing? How will you measure (heh) if it’s working?
Employees will likely have common concerns: Explain how metrics are collected and used, who sees what, and where information is stored.
Feedback paths: Explain how people can share their thoughts and how you will address their feedback.
Make metrics accessible to the people who the metrics are for and about. Examples include:
Dashboards, spreadsheets, observability tools, and other data visualization tools are accessible for all employees whose activity flows into them, even if only in aggregate.
Managers have access to data about their groups in aggregate.
Don’t get annoyed if people ask about these things. Approach what may look/feel like resistance to this change with curiosity. This guide has a lot of tips for how you can work with your teams to address their concerns, productively, together.
Step 5: Use your metrics, visibly, and make them make sense
This step is really important, and in my experience the step where leaders run into issues and employees get frustrated. After all, if you ask your employees to track a bunch of data for you and don’t end up using it, you’re wasting people’s time. So:
Use your metrics. Visibly.
And explain how you do that, what the flaws and gaps are, and how you will supplement or work around them. Examples for using metrics visibly:
Show them in every company/department/team meeting. Talk about trends and how you’re adjusting plans based on these metrics.
If you have a special set of metrics about your organization, show how you use them. As an engineering executive, I typically have to present metrics to executive teams and the company’s board of directors. After these presentations, I give them to my teams too. Team members usually really appreciate this, as it helps them understand how I talk about the organization, and where we’re at as a whole.
When you make decisions, explain which metrics you used to inform said decision.
Make sense of the metrics by adding context and a narrative
Metrics are only as useful as the context they’re presented in. I recently worked with a VP Engineering on a presentation to their executive team and board of directors. The VP had included several neat-looking charts, but needed to add more context that the data alone didn’t provide -
Context is critical to take your audience from seeing a bunch of numbers, to actually getting meaning
Context includes complementing your metric(s) with information like:
What does this metric mean for you and your organization?
Is this number good or bad? How does it compare to historical data?
Is this what we expected? Or is this a surprise?
How does it fit into historical data and trends? What direction are we expecting for the future?
What are we doing about this? Are we taking action to improve? Or are we accepting that something isn’t ideal because we have other, more important priorities?
Remember, the key is not just choosing any metric because everyone else does. The transformational power of metrics lies in selecting those that align with your organization's goals and help your team improve.
To unlock this power, start with understanding the underlying problems these metrics are meant to address. The right metrics can provide invaluable insights when you couple them with transparent communication, team involvement, and a clear understanding of the metrics’ purpose and limitations. That’s when they’ll become the basis for improved conversations, joint learning, and to make better decisions.
Stay tuned for more insights in this series, where we'll continue to explore the dynamic world of engineering metrics. In the next post, I finally share the answer to one of the questions I get asked the most:
Lena, what engineering metrics do you like? And what engineering metrics should I use? Read my answer here.
This is the latest part of my guide where I help you use metrics successfully with and in your team(s), based on practical tips and lessons learned in my engineering leadership career. Further parts of this series cover:
