The Role of Metrics in Crafting Engineering Culture
Tl;dr & Executive Summary
Align metrics with the goals and problems specific to your team or organization. Use them as a means to drive positive change, not just as arbitrary markers of performance.
Metrics create incentives that shape behavior and, ultimately, the culture of your team, and may lead to undesired consequences. Therefore, when choosing metrics,
Avoid overemphasizing one category; balance effectiveness and efficiency.
Use both leading and lagging indicators for a complete picture.
Consider the complexity of human and technical systems when selecting metrics.
Many leaders ask me: “What metrics should I use to measure my engineering teams and organization? What metrics do you use, Lena?” And there’s lots of advice out there on engineering metrics, from “the 10 metrics every engineering team should use”, to frameworks like SPACE and DORA. It’s tempting to measure as much as you can, and may even give you the impression that you feel like you’re on top of things.
But using metrics just for the sake of it, or because your boss tells you so and you hear from others that it’s the right thing to do, will do more harm than good. Metrics are important, but they won’t matter if you’re not using them effectively.
To be clear, I think these frameworks and lists of metrics can be useful. But they’re also a shortcut.
Metrics are a tool to solve a problem;
they’re not the solution on their own.
That’s why I want to give you a tool for how to choose metrics and actually make them work for your team. You’ll learn about how to think about metrics, and what metrics I’ve found useful.
What metrics can be useful for
Metrics help you:
Manage up: Help your boss and executive team understand what your team is doing, how, and the impact of your work. Metrics support good business cases and increase the visibility of your team’s work.
Manage your team: Use metrics to find out how your team is performing and where you need to make investments.
Manage your stakeholders: Being able to demonstrate e.g., what percentage of your team’s capacity has been going into Product-led work, or how many bugs are escaping, is usually a much more useful conversation than having to discuss the same based on gut feel.
Manage yourself: Having better visibility through metrics also means that you’re able to more quickly spot when things are going wrong, and course-correct.
Why metrics matter more than you think
Metrics create incentives.
Incentives create behaviors.
Behaviors create culture.
Metrics create incentives to act a certain way, or to stop doing something. They point towards what we’ll be paying attention to as leaders: They signal “this area is important”, which leads to our teammates paying more attention to it too.
Incentives create behaviors. This is called Measurement reactivity - when people know they’re being measured on something, it will alter the way they behave - and we can use this to our advantage with our teams, shaping how we act by choosing how we’ll measure ourselves together.
Behaviors create culture. Culture is the behavior we reward and punish. Every piece of feedback we give, every promotion, every meeting invite - all create culture, whether you like it or not.
The metrics you choose are going to craft the culture of your organization. That’s why not just what you measure, but how you use these metrics, is so important.
The right metrics for your team
Don’t overindex on one category of metrics. Measure effectiveness and efficiency, don’t overindex on only measuring in one of these areas. Even if your company is doubling down on efficiency right now, you should still measure effectiveness as well - even if only to keep an eye out for making sure that your effectiveness doesn’t go down the drain.
Balance metrics to avoid setting “wrong” incentives. When I set metrics, I ask my team, “What if we only did this one thing, and did it to the extreme?” - to think through what unintended consequences a metric may lead to, and balance it out. One example: Cycle time, escaped defects, and planned-to-done ratio balance each other very well. If you only focused on short cycle time, you’d likely end up with more escaped defects; or if you only focused on a high planned-to-done ratio, your cycle time may be skyrocketing.
Do justice to the complexity of technical and human systems. Many teams use metrics to try to answer questions like “why is this project taking so much longer than we thought?” Questions like this one require complex and nuanced answers and a mix of a few metrics and context, such as: What happened that isn’t visible at a glance from the data? Maybe there were staffing changes on the team, new requirements came in, success criteria became clear late in the planning process, or something was overlooked in planning?
Mix leading and lagging indicators; don’t just measure what you’ll only know late in the game: We only know whether we’re achieving our goal when we reach it (or don’t), so goal accomplishment is a lagging indicator. On the flip side, deployment frequency is a leading indicator: It shows a trend very quickly. A mix of leading and lagging indicators helps you spot trends early on and adjust accordingly.
Start with what you can know and can impact easily. Begin with what you can measure, and invest in areas that are harder to measure later.
Teams need to know what they’re being measured on. Share metrics about your team with your team.
Few, shared, reviewed, and used metrics are better than lots of metrics you don’t use. Metrics are best when your team knows what these metrics are and you use them regularly together: Review them in your retrospectives, and invest in getting better. That’s when metrics help you impact change.
Metrics are signals, not targets.
They’re lamps illuminating your path,
not the light to move towards.
So what metrics should I use?
Whenever you’re asked to provide more metrics, your first priority is to understand the goal. Ask:
What would you like to understand better about our team?
What problem are you trying to solve?
Who’s the audience? Who do you want to provide this information to? What does this person care about?
Cases where metrics won’t help you
Metrics have their limits and can cease to be useful very quickly when you’re not selecting and using them properly:
Metrics are about your team, but without them. People need to know what they’re being measured on - that’s the least we can do not just from a moral standpoint, but also because keeping metrics a secret means you’re missing out on all the potential for using the metrics to create the culture you want and need.
The metrics measure individual performance. Great software and products are built by stellar teams, not by individual heroics. Metrics like work hours, lines of code, or individual pull requests entirely miss this point.
You don’t turn the insight from metrics into action. If you and your team review every week that your reliability is low, but you never prioritize fixes, all you’re going to get is a frustrated, demotivated team. If your company runs an employee engagement survey every quarter, but but doesn’t:
Close the feedback loop by communicating high-level findings back to the employees;
Define concrete follow-up actions to address employees’ feedback; and
Prioritize and execute on those actions;
You’re not actually using the insight the survey gives you. Even worse, you’re going to demotivate your employees. Typically, in such situations, engagement and survey participation rates drop.
In addition, metrics are only a part of a bigger story about what’s going on:
Metrics aren’t as objective as they seem: In choosing and using metrics, you make choices that reflect your and your organization’s goals and biases:
What you measure and don’t: Do you only focus on technical metrics? Or do you also measure “soft” areas like employee satisfaction? What you choose (not) to measure signals what you have (not) been paying attention to.
When and how you communicate these metrics, and to whom: Do you share those metrics with your team? Do they have direct access to them?
What you prioritize based on the insight: Do you continue with business as usual, or make changes with your team based on what you’re learning from your metrics?
Metrics are meaningless without context. I’ve been working with a team recently on an engineering department presentation to their company’s board of directors. My most-asked remark on the slides was: “Is what you’re describing here a good thing? A bad thing? How does it compare to the previous year? And what are you doing about it?” Metrics only become really valuable when contextualized with:
A baseline to compare with, like metrics for the team/organization in a previous quarter, or your entire organization or company. In addition, it always helps to add
Qualitative context: The number of bugs increasing can be a bad thing - or a good thing if your company just introduced a new reporting tool and you’re now catching them sooner. Your team’s cycle time increasing can be a bad thing - or a good thing if your team has been focused on interviewing new team members and spent the planned amount of time on interviews.
Using metrics the right way can elevate your team, business and culture
Figuring out the metrics which make the most sense for you and your team's needs, while bearing in mind their limitations, provides a great starting point for figuring out what’s driving success, and where there’s room for improvement. Done the right way, teams will embrace metrics, not push against them.
More in this series:
Why your boss wants metrics: The role that metrics play for leaders at different levels, and how to use them as a tool for alignment and accountability with your boss and direct reports.
How to choose and use metrics: Learn how to select the right metrics for your organization, implement them effectively, and go from mere numbers to actionable insights, to create a culture of data-driven decision-making.
