“Am I good or what?” The question was, of course, rhetorical, I was alone in my office. I
Photo Courtesy of: https://www.flickr.com/photos/bike/
couldn’t help it though, I was pleased as punch and nothing was going to ruin my great mood.
“Or what?…”
Not even an 800-pound invisible gorilla.
“Go away, Hogarth, you can’t ruin my mood today. I’m on top of the world.”
Ignoring me, Hogarth ambled into my office. Spotting my new fichus, he plopped himself down and tore a branch from the tree. Around a mouthful of leaves, he asked: “So, agile adoption going well?”
His question instantly dispelled my annoyance at his assault on my plant. “Yes, yes it is.” I turned the monitor around so he could see. “Just look at these velocity trends! Every team is hitting or exceeding their velocity targets and we’re only three sprints in. It’s absolutely fantastic!”
Hogarth leaned in and intently studied the flat screen display. “Impressive. That’s got to be one of the fastest velocity growths I’ve ever seen. What did you do differently this time?”
“Hey, I’m just good. Awesome training, great coaching. Oh, and I bet the incentive program really helped out.”
Hogarth cocked his head to the side, “Incentive program?”
I leaned back smugly, “Oh, yeah. If the teams hit the velocity goals then they get a cash bonus and we’re going to hold a huge party at Humdingers Resort.”
Hogarth nodded, making some appreciative sounding grunts. I’d finally gotten to him. He was speechless.
After a long gaze at the monitor, he turned to me and gave one of his brilliant white smiles. “Sounds like a real Goodhart moment.”
It was my turn to cock my head in confusion. “A Goodhart moment?”
My gorilla nodded at me, “Uh huh. A well known British economist has an economic concept named after him. The layman’s version of Goodhart’s Law states, ‘When a measure becomes a target, it ceases to be a good measure.'”
I looked at Hogarth. I looked at the velocity reports. I looked at Hogarth. “You mean…”
He nodded, “Yup, those velocity metrics are about as useful as wheels on a speed boat. They look good spinning, but they are not really doing anything.”
Metrics- Can’t live with them, can’t live without them.
“Metrics are not bad, managers using metrics improperly is bad.” was a quote I sent out on Twitter (@JBC_GC) during Agile Open Northwest 2017. The session was “Why do metrics get a bad rap?” and it was a lively conversation with some surprising outcomes.
I’ve had a lot of success with the healthy use of team metrics. I’ve used the afore mentioned Goodhart’s Law as a conversation starter on the good use of metrics many times. And despite my success with team metrics I had never really articulated, to myself or others, what the disconnect between team metrics and management’s desire to set goals on those team metrics was. Turns out it is rather a simple thing.
Managers don’t actually care about metrics. They care about success.
Managers want to know if the product will ship on time or with the promised features or with the promised value, or all of the above. They end up using team metrics because it’s all they have. And in the classic square peg in a round hole scenario, they were hitting the square peg with the hammer to make it fit.
And how’s that worked out for us? Even in the well run agile project, the ability to tie team estimates and metrics to actual shipping dates is a highly mixed bag. We are horrible at estimating, to the point that the #NoEstimate movement is gaining traction through being right. This, however, is not a rant about how bad we are at estimating, using no estimates or even bad metrics. It is instead a look at what we can do about this disconnect between Team Metrics and Management need.
Step 1: Recognize managers need better forecasts, not better metrics.
Step 2: Stop using Team Metrics for forecasting.
Step 3: Give managers the tools to let them forecast.
Step 1: Recognize managers need better forecasts, not better metrics.
The best use of metrics is “as a lagging indicator of if we might want to talk to the team and see if they need help.” This is the coaching advice I give to management and teams, usually shortly after quoting Goodhart’s law. Metrics will not tell you when a team will get done without the risk that the metric will run afoul of the wisdom of Mr. Goodhart.
Just as you want to know if you should pack an umbrella tomorrow (or wear sunglasses, or get your snow blower ready), management wants to know if the project will be successful. A totally and perfectly valid request. Only you don’t use team metrics for this. Even when metrics are used as the source data, they are still being used to forecast. We look at velocity trends to estimate when a team will be done. That’s forecasting, not metrics.
Step 2: Stop using Team Metrics for forecasting:
Performance based on incentives doesn’t work. When we try and use metrics to drive performance we get Goodhart’s Law. The example I like to give comes from the book Freakanomics.
To summarize: India was having a problem with cobras. To address the problem the government came up with a great idea. They would give a bounty on every cobra head turned into the government. The program was a rousing success, just not in the way the government intended. People started raising cobras just so they could collect the bounty. In short order, India’s cobra problem was even larger than before the bounty was put in place.
If you tell a team they must make a March 15th release date. They will very likely hit that date or come really darn close. And then the company ends up dealing with bugs for the next five years. “When a measure becomes a target, the measure is no longer valid.” If you’re struggling to get your message heard, you might try showing them Dan Pink’s RSA Animate video to reinforce that knowledge skill teams don’t work best from incentives, they work best from Autonomy, Mastery, and Purpose.
Of course, not everyone is going to listen, even when they pay us for our advice. Which is why I advocate limiting the metrics used and make them interlocking. Game one metric and the others will react in the negative. My four metrics are based on Jeff Sutherland’s three recommended metrics and a fourth I’ve had a lot of success with. They are:
- Cycle Time
- Escaped Defects
- Happiness Metrics
- Planned to Done Ratio
If you game cycle time so it’s really short, quality will almost certainly suffer. Let quality slip and you see an increase in cycle time and escaped defects. If planned to done is too high, then quality is probably suffering. And if metric 1, 2 and 4 are all really great and happiness is suffering then you have a strong indicator that you’re burning out your team.
Step 3: Give managers the tools to let them forecast.
This one is really easy. Stop reading my blog and go to FocusedObjectives.com. Troy Magennis has crossed the streams of mathematics, classic project management forecasting and agile to come up with a tool that allows managers to forecast when a team(s) or feature(s) will either be done or if it will hit the desired schedule.
I’ve seen this tool in action and heard stories from several users of Troy’s Forecasting approach. With this tool, the managers no longer need detailed metrics from the team. They can build forecasts based on as little as a half dozen data points, which can even be made up for initial forecasts. With just cycle time data points, you can run massive Monte Carlo simulations to get 80-90% accuracy on your forecasts (note, anyone who claims 100% accuracy is also gaming the system, or in denial).
So keep team metrics focused on improving the team. Give management their own tool for forecasting schedules and/or capacity. Then watch as the results of this is the teams getting, even more, work done, with greater predictability, and greater happiness.