The Middle Manager’s Guide to Engineering Metrics (2): How to choose and use metrics
This article is part of my series “The Middle Manager’s Guide” with practical tips and lessons learned in my engineering leadership roles to help you thrive as a current or aspiring senior leader. It was co-written with Maggie Litton, Director of Engineering. More in this series:
The Role of Metrics in Crafting Engineering Culture: How metrics shape behavior and create culture, and how you can set balanced, effective metrics that don't just measure, but also matter.
Why your boss wants metrics: The role that metrics play for leaders at different levels, and how to use them as a tool for alignment and accountability with your boss and direct reports.
Tl;dr & executive summary
For metrics to be truly useful, follow these steps:
Understand the Problem: Before selecting metrics, clarify the problem you're trying to solve. Ask critical questions to understand what outcomes you're aiming for and who the stakeholders are.
Decide if Metrics are the Solution: Evaluate if metrics alone can solve the issue or if there are underlying structural, cultural, or procedural problems that need to be addressed.
Build Trust: Ensure transparency when introducing metrics.
Choose Metrics Wisely: Don’t try to measure everything just for the sake of it: Allow employees to pick metrics that are directly relevant to them. Opt for metrics with easily gatherable data. Involve your Direct Reports, providing them with the context and expectations to come up with useful metrics for and about their teams.
Roll Out Metrics with Care: Clearly communicate the intention and context behind the metrics. Create feedback channels and make the metrics accessible to the relevant people.
Use Metrics Visibly: Keep metrics front-and-center in discussions and decision-making processes to show that they are being actively used.
Add Context to Metrics: Metrics are most valuable when presented with context that adds meaning to them.
Metrics are a really useful tool to get the visibility you need as a leader. As seen in the first part of this series on engineering metrics, the leaders at different levels in your organization bring different contexts, purviews, and goals to metrics. As a middle manager, a big part of your role is to balance those. Now you understand everybody’s context, how do you actually pick metrics? And how do you make them work?
Leaders usually ask me one of two questions:
Help, my boss is asking me to come up with metrics! You’ve been tasked with “coming up with metrics” for your team(s) (admittedly, I’ve also been the boss in this scenario). Or -
I want to use metrics for my team(s), what do I do?
The good news is: The steps to make metrics work for you and your team(s) are the same in all cases. In our joint 35+ years of leadership experience, Maggie and I found that the following steps are critical for making sure metrics are actually useful, no matter what level you’re at. And first, let’s get one thing out of the way:
It doesn’t matter what metrics you use
Metrics are a tool to solve a problem; they’re not the solution. The metrics you use will only be as useful as your clarity on the problem you’re trying to solve with them.
Metrics have been a topic of hot debate in tech for a really long time. In this industry where pattern-matching is almost the norm and being on top of the latest trends is seen as a virtue, it’s easy to feel pressure to follow the latest trends. But:
What works for one organization will not work for yours without adjustments.
Take this example: I currently work with two software companies that are similarly sized and both very successful in their respective fields. One of them has been in business for 25 years and is just introducing unit tests (yes, you read correctly); the other company is fine-tuning its continuous deployment setup.
It does not matter what metrics you use. It doesn’t matter if you pick DORA, SPACE, DevEx, agile metrics, employee engagement, or some handcrafted metric that’s pulling data from two databases and a spreadsheet.
Metrics only matter if they’re aligned with your organization’s goals and you’re actually using them.
A step-by-step-guide to making metrics work and creating a data-driven organization
1. Understand the problem you’re trying to solve
Whenever I speak with leaders who tell me they want to introduce certain metrics, I like to dig into what they’re trying to achieve. With no exception, it becomes clear very quickly that their actual concern goes far beyond a specific metric, and is instead:
We’ve hired a lot of people, why are we not getting as much done as I’d expect?
Why is Project X taking so long?
Why did we just have another incident in [area]?
What are we doing to fix $problem?
I’m getting questions from their fellow executives or the board that I don’t have answers to.
I feel like they don’t have a good grasp on a certain area.
Case study: “Story Points”: Recently, I worked with an engineering executive team that wanted to start measuring story points across their teams. As I asked more about their goals, it became clear that they had concerns about a project that was supposed to be released months ago, but kept getting delayed. In the end, the executives decided that the actual steps they needed to take were to work with the team directly and help them scope out the project more clearly, as well as on their planning, communication, and stakeholder management. Story points wouldn’t have helped here (side note: Story points are also not a metric, but more on this below).
This means: No matter who asks you for metrics, your first step should be to understand the problem that needs to be solved. Get curious, and, for as long as you can, focus the conversation on really understanding the problem. Ask your boss (or yourself) questions like the following; you don’t always have to ask all these questions, I’m listing them here to give you ideas of what can be helpful to understand:
Define the problem
What’s the problem you’re trying to solve? What outcome(s) are you trying to achieve? - Some examples:
Are you trying to learn?
Are you trying to measure an effect or progress? Maybe you’ve made a change in your architecture and you want to know if it reduced user-experienced latency as much as you hoped it would.
Are you trying to gain support for a project? Or maybe you want to highlight the impact of toil work on engineering opportunity costs?
Is your goal to satisfy a directive from your boss? Or give them better visibility into your team(s)?
Do you want to improve in a specific area like reliability, quality, effort, cost-effectiveness, efficiency,...?
What makes you believe these metrics will help solve those problems?
Explain the connections between the team and the organizational metrics that are being collected. This is especially important if your boss is relying heavily on proxy metrics or composite metrics that can mask how your team contributes to those numbers.
What information is missing? What would be helpful/persuasive to know more about?
Know the audience
Goal: Understand the stakeholders better, so you can identify what metrics best address their concerns.
Who needs the solution to this problem? Who’s the audience for these metrics?
How will the metrics be used? By whom?
Obtain and update data
Goal: You likely don’t have all required data at hand yet, and some requests can take a lot of work to get this data together.
How quickly do you need this information?
How often does the data need to be updated?
How high a priority is this? [If needed] What do you want to deprioritize to get this done?
What time and resources can we use for this? This is important if obtaining the data requires automation or telemetry investments.
Involve the team
Goal: In the spirit of transparency and making metrics useful, see if you can make this a collaborative effort.
How can we involve the teams in coming up with metrics?
How do you want to handle feedback from team members about this?
2. Decide if metrics are (part of) the solution
In my experience, many problems actually require a dual approach with a set of actions run in parallel:
Introducing metrics. I’ve found that it’s rare that a single metric can answer a complex organizational question.
Addressing cultural, structural, and procedural issues, such as poor communication and stakeholder management, dysfunctional team structures, weak delivery practices like low predictability of delivery, and poor expectation management.
Case study: In the case of the aforementioned “Story Points” case study, the engineering team took the following steps in parallel:
Introducing metrics: The team started using story points as an internal tool for improving their planning practices. They also realized that lack of focus was an issue and introduced a WIP (work in progress) limit.
Addressing process issues: The team made a lot of changes to their delivery processes, such as working more closely with stakeholders, spending more time on planning, and breaking work into smaller chunks.
3. Lead with trust and transparency
Show and earn trust with transparency: Even when you as a leader introduce metrics with the best intentions, beware that many employees have seen metrics weaponized against teams and individuals in the past. They’ll likely have concerns: Leaders are saying one thing now, but if leadership or direction changes, is this going to still remain the case?
In our experience, it’s really helpful to be transparent about wanting to introduce metrics early on. When you share with your team that you plan to introduce metrics, speak to them about:
Your intentions: What are you trying to achieve? This may seem obvious to you; assume that not everyone understands the business or specific metrics well yet.
Context: Where is this coming from?
Speak to common concerns that employees will likely have: Explain how metrics are collected and used, by whom, who can see what information (will the engineers have access to this data?), and where information is stored.
Create feedback paths: Explain how people can share their thoughts and how you will address the feedback.
Don’t get annoyed if people ask about these things.
4.1 Choose metrics wisely
Making smart choices about which metrics you pick is a game changer for really making metrics work well; read below under 4.2 about how to involve your line managers and other direct reports.
(A) Let your employees decide what they want to track for themselves. Let folks who report to you come up with their own metrics, even if you don’t fully buy into them. In the last article, I showed the difference between metrics for and metrics about teams. Let your teams pick the metrics that are for them - after all, those metrics will be most useful when they help your employees actually learn and improve. How to do this:
(B) Use metrics for which data is easy to gather. Don’t make people jump through unnecessary hoops for things they don’t care about. There’s a balance between having 15 mandatory JIRA fields to get as much consistent data as possible, vs requiring none and getting no useful data. There can be cases when you want to use metrics for which data is harder to gather; keep those to a minimum.
(C) Don’t use metrics that aren’t actually metrics. Many engineering executives want to use numbers that may look like metrics but aren’t in all cases. Common examples include:
Numbers that are useful metrics at the team level, e.g., velocity, including story points and t-shirt sizes. These are tools that are super useful for teams internally to help them get better at planning their own capacity and as a forcing function to break down work into smaller chunks. Do not use them across teams: They’re relative measures and not comparable across teams.
Numbers that aren’t useful at all, like lines of code and similar output measures. Sometimes, the most productive thing a team can do is delete old code. Don’t use such output measures.
(D) Don’t be afraid to keep multiple sets of metrics: Some for your own managers, some for folks who report to you, and some for specific teams. Not all metrics will be relevant to everyone at all times, and that’s okay.
(E) Be willing to accept a metric that you find sub-optimal, especially if it buys you relationship capital or if you can balance it with other metrics. These are cases where it can help to take the pragmatic road and use a metric, even if you don’t think it will be very meaningful:
Case Study: “Time to first Pull Request”: A company was hiring a lot, and their leadership team wanted to know how quickly new hires get productive, measured by “time to first pull request/merge request (PR/MR).” In isolation, this metric, like most, has limited value:
A new hire’s first PR/MR may be a trivial exercise aimed at giving them a sense of how the build pipeline works.
The onboarding process required all new hires to make a small PR within their first week; this doesn’t mean they’re capable of working independently in a codebase.
Rather than strongly objecting to a metric your leadership is fond of, engineering managers added an additional metric like “self-reported feelings of productivity” to their onboarding survey, and and “time-to-10-th PR/MR.”
Case Study: “Cyclomatic complexity”: I once worked with a colleague who was eager to measure cyclomatic complexity in a codebase. I didn’t think this metric would translate well for leaders outside of engineering, but my colleague was very excited about it. I decided that including this metric in our charts wasn’t going to hurt anything as long as we also had another metric that was less esoteric.
(F) Only gather as many metrics as you’re actually able to use.
Metrics aren’t a pokémon-style “gotta catch ‘em all”-exercise: Resist the temptation to gather a ton of metrics. Focus on metrics that you’ll actually use.
(G)
I often see leaders trying to measure everything they can. Not only does this become very expensive very quickly (in terms of time, tooling cost, storage space, manual labor for data entry, data analysis, etc.), but it also very quickly defeats the purpose.
4.2 Involve your direct reports in coming up with metrics
As a middle manager who was asked by your boss to come up with metrics, you’ll next need to involve the line managers who report to you. Here’s how to do this well:
Share context! Help your direct reports understand the purpose of this exercise. Use the list in “Step 1: Understand the problem you’re trying to solve” in this article as a checklist for what to cover, from the problem you’re trying to solve, to the time and resources they may use for this.
Set expectations with your line managers. Not all line managers have much experience using metrics, and some may find it difficult to make a connection between their team and the business through metrics. Share what you need from them for:
Standardization: Your line managers need to know when all teams must measure things in the same way, usually when higher-level bosses want to view aggregate data.
Targets: Do you have a target in mind? Are you looking to measure trends or absolute values?
Customization: Your direct reports will be more engaged if they also have room for some autonomy and customization, such as if there’s space for them to add metrics that their team also cares about, and not just executives.
Numbers that you and your line managers can easily explain. Avoid overly complicated concepts or statistics that will be hard to translate to executives or people in other departments, like the “cyclomatic complexity” example above. Metrics like “developer satisfaction”
Metrics that convey the connection between engineering work and business value. Talk with your line managers about how their teams contribute to business value; as a middle manager, you likely have a broader context and can help them make this connection. Steer them away from only using team-level metrics like velocity that are fine to use for teams internally but usually don’t fulfill middle managers’ and executives’ needs.
Line managers need to be able to use metrics - it’s an important skill, especially if they aspire to higher-level roles.
5. Roll out metrics with clarity
As you roll them out, speak more about the points shared above in 3. and address the common concerns your employees will likely have. You can also find a communication structure and further ideas for handling such a change in this article about managing change.
Make metrics accessible to the people who the metrics are for and about. Examples:
Dashboards, spreadsheets, observability tools, and other data visualization tools are accessible
Managers have access to data about their groups in aggregate. This should include that managers have access to their group’s budget (!) and data like employee engagement survey results.
6. Use metrics visibly
This one is really important, and in my experience, it’s the step where leaders become forgetful, slack off, or run into issues, and employees get frustrated. After all, if you ask your employees to track a bunch of data for you and don’t end up using it, you’re wasting people’s time. So:
Use your metrics. Visibly. And explain how you do that, what the flaws and gaps are, and how you will supplement or work around them. Examples for using metrics visibly:
Show the metrics in every company/department/team meeting. Talk about trends and how you’re making decisions and adjusting plans based on these metrics.
If you have a particular set of metrics about your organization, show how you use them. As an engineering executive, I typically have to present metrics to executive teams and the company’s Board of Directors. After these presentations, I present them to my teams too, and explain what the metrics are for, and how the Board uses them. Team members usually really appreciate this, as it helps them understand how I talk about the organization and where we’re at as a whole.
When you make decisions, explain which metrics you used to inform said decision. This also helps cultivate a culture of data-driven decision-making.
7. Go from metrics to meaning with context
Metrics are only as valuable as the context they’re presented in.
Don’t rely on quantitative data alone:
Make sense of the metrics by adding context and creating a narrative.
I recently worked with a VP of Engineering on a presentation to their executive team. The VP had included charts, but as I reviewed the presentation draft, I added several feedback points about missing context that I’m sharing with you here -
Whenever you review or present data, include:
The story: What's the story you want to tell, and what do you want people to take away from this?
An explanation of the metrics, depending on your audience: Don’t assume that everyone knows what e.g., cycle time is; when in doubt, include a short explanation, such as: Cycle time measures how long a work item takes from our engineering team planning to take it on, to being done. [Your mileage may vary.]
Meaning, implications: What does the data mean for you and your organization?
Event context: What important events happened in the last [timeframe] that are or will be impacting this data?
Judgment: Are the metrics you're presenting good, bad, desired, undesired? How do they compare to historical data?
Alignment with expectations: Is this what you expected? Or is this a surprise? - This is an important point: After all, it’s a very different situation if you were surprised by e.g. an increase of delays in addressing bugs, or if it was expected because the company was moving to a new ticketing system.
Future trends: What direction are we expecting for the future?
Action: What are you doing about this? Are you taking action to improve in these areas? Or are you accepting that something isn’t ideal because you have other, more important priorities?
Your opinion: What’s your opinion on this data? Especially as a higher-level leader, you’ll likely be expected to have an opinion about what you’re sharing.
In short: Make data the supporting actor of your story, not the only story - the story is context woven together with your data points into a narrative.
In a world awash with data, the skill isn't just to collect metrics but to discern the narrative behind the numbers.
Metrics are not just a crucial part of our jobs as managers and leaders: used wisely, they are a catalyst for transforming organizations.
Metrics are not a monologue; they're a conversation between you, your team, and the broader organization. Invite everyone to participate in this conversation, ensuring that your measures are not only transparent but also embraced by those who will be measured. Your aim should be metrics that are not just 'followed' but lived—integrated into the very heartbeat of your team's daily operations.
As you adopt, adapt, and advance how you and your teams use metrics, ensure that they continue to serve as the compass that navigates your team toward achieving your goals.
Metrics, at their best, are not just a tool but a language that helps articulate your organization's unfolding story of success. In the next part of this series, I’ll share with you what metrics I use in my work as an engineering leader.
