Metrics: Third Rail of Agile Adoption

“Am I good or what?” The question was, of course, rhetorical, I was alone in my office. I

Photo Courtesy of: https://www.flickr.com/photos/bike/

Photo Courtesy of: https://www.flickr.com/photos/bike/

couldn’t help it though, I was pleased as punch and nothing was going to ruin my great mood.

“Or what?…”

Not even an 800-pound invisible gorilla.

“Go away, Hogarth, you can’t ruin my mood today. I’m on top of the world.”

Ignoring me, Hogarth ambled into my office. Spotting my new fichus, he plopped himself down and tore a branch from the tree. Around a mouthful of leaves, he asked: “So, agile adoption going well?”

His question instantly dispelled my annoyance at his assault on my plant. “Yes, yes it is.” I turned the monitor around so he could see. “Just look at these velocity trends! Every team is hitting or exceeding their velocity targets and we’re only three sprints in. It’s absolutely fantastic!”

Hogarth leaned in and intently studied the flat screen display. “Impressive. That’s got to be one of the fastest velocity growths I’ve ever seen. What did you do differently this time?”

“Hey, I’m just good. Awesome training, great coaching. Oh, and I bet the incentive program really helped out.”

Hogarth cocked his head to the side, “Incentive program?”

I leaned back smugly, “Oh, yeah. If the teams hit the velocity goals then they get a cash bonus and we’re going to hold a huge party at Humdingers Resort.”

Hogarth nodded, making some appreciative sounding grunts. I’d finally gotten to him. He was speechless.

After a long gaze at the monitor, he turned to me and gave one of his brilliant white smiles. “Sounds like a real Goodhart moment.”

It was my turn to cock my head in confusion. “A Goodhart moment?”

My gorilla nodded at me, “Uh huh. A well known British economist has an economic concept named after him. The layman’s version of Goodhart’s Law states, ‘When a measure becomes a target, it ceases to be a good measure.'”

I looked at Hogarth. I looked at the velocity reports. I looked at Hogarth. “You mean…”

He nodded, “Yup, those velocity metrics are about as useful as wheels on a speed boat. They look good spinning, but they are not really doing anything.”

 

Metrics- Can’t live with them, can’t live without them.

“Metrics are not bad, managers using metrics improperly is bad.” was a quote I sent out on Twitter (@JBC_GC) during Agile Open Northwest 2017. The session was “Why do metrics get a bad rap?” and it was a lively conversation with some surprising outcomes.

I’ve had a lot of success with the healthy use of team metrics. I’ve used the afore mentioned Goodhart’s Law as a conversation starter on the good use of metrics many times. And despite my success with team metrics I had never really articulated, to myself or others, what the disconnect between team metrics and management’s desire to set goals on those team metrics was. Turns out it is rather a simple thing.

Managers don’t actually care about metrics. They care about success.

Managers want to know if the product will ship on time or with the promised features or with the promised value, or all of the above. They end up using team metrics because it’s all they have. And in the classic square peg in a round hole scenario, they were hitting the square peg with the hammer to make it fit.

And how’s that worked out for us? Even in the well run agile project, the ability to tie team estimates and metrics to actual shipping dates is a highly mixed bag. We are horrible at estimating, to the point that the #NoEstimate movement is gaining traction through being right. This, however, is not a rant about how bad we are at estimating, using no estimates or even bad metrics. It is instead a look at what we can do about this disconnect between Team Metrics and Management need.

Step 1: Recognize managers need better forecasts, not better metrics.

Step 2: Stop using Team Metrics for forecasting.

Step 3: Give managers the tools to let them forecast.

 

Step 1: Recognize managers need better forecasts, not better metrics.

The best use of metrics is “as a lagging indicator of if we might want to talk to the team and see if they need help.” This is the coaching advice I give to management and teams, usually shortly after quoting Goodhart’s law. Metrics will not tell you when a team will get done without the risk that the metric will run afoul of the wisdom of Mr. Goodhart.

Just as you want to know if you should pack an umbrella tomorrow (or wear sunglasses, or get your snow blower ready), management wants to know if the project will be successful. A totally and perfectly valid request. Only you don’t use team metrics for this. Even when metrics are used as the source data, they are still being used to forecast. We look at velocity trends to estimate when a team will be done. That’s forecasting, not metrics.

Step 2: Stop using Team Metrics for forecasting:

Performance based on incentives doesn’t work. When we try and use metrics to drive performance we get Goodhart’s Law. The example I like to give comes from the book Freakanomics.

To summarize: India was having a problem with cobras. To address the problem the government came up with a great idea. They would give a bounty on every cobra head turned into the government. The program was a rousing success, just not in the way the government intended. People started raising cobras just so they could collect the bounty. In short order, India’s cobra problem was even larger than before the bounty was put in place.

If you tell a team they must make a March 15th release date. They will very likely hit that date or come really darn close. And then the company ends up dealing with bugs for the next five years. “When a measure becomes a target, the measure is no longer valid.” If you’re struggling to get your message heard, you might try showing them Dan Pink’s RSA Animate video to reinforce that knowledge skill teams don’t work best from incentives, they work best from Autonomy, Mastery, and Purpose.

Of course, not everyone is going to listen, even when they pay us for our advice.  Which is why I advocate limiting the metrics used and make them interlocking. Game one metric and the others will react in the negative. My four metrics are based on Jeff Sutherland’s three recommended metrics and a fourth I’ve had a lot of success with. They are:

  1. Cycle Time
  2. Escaped Defects
  3. Happiness Metrics
  4. Planned to Done Ratio

If you game cycle time so it’s really short, quality will almost certainly suffer. Let quality slip and you see an increase in cycle time and escaped defects. If planned to done is too high, then quality is probably suffering. And if metric 1, 2 and 4 are all really great and happiness is suffering then you have a strong indicator that you’re burning out your team.

Step 3: Give managers the tools to let them forecast.

This one is really easy. Stop reading my blog and go to FocusedObjectives.com. Troy Magennis has crossed the streams of mathematics, classic project management forecasting and agile to come up with a tool that allows managers to forecast when a team(s) or feature(s) will either be done or if it will hit the desired schedule.

 

I’ve seen this tool in action and heard stories from several users of Troy’s Forecasting approach. With this tool, the managers no longer need detailed metrics from the team. They can build forecasts based on as little as a half dozen data points, which can even be made up for initial forecasts. With just cycle time data points, you can run massive Monte Carlo simulations to get 80-90% accuracy on your forecasts (note, anyone who claims 100% accuracy is also gaming the system, or in denial).

 

So keep team metrics focused on improving the team. Give management their own tool for forecasting schedules and/or capacity. Then watch as the results of this is the teams getting, even more, work done, with greater predictability, and greater happiness.

Just say “No” to Gorilla Debt in your Teams

By ccPixs.com

Debt Free by ccPixs.xom

You’d think with the amount of time I spend hitting my head on my desk it would have a permanent dent by now.

My most recent forehead abuse was brought short by the one thing I hated more than coaching a team that thought they we’re already perfect. The deep, rumbling bass of my personal gorilla cut through the sound of my head hitting the desk. “The SPFA called, they are filing a restraining order against you for furniture abuse.”

I blinked and looked up at the hulking form of my conscience in physical form. Why couldn’t I have a little cricket like Pinocchio? It would be so much easier to lock him in a jar and toss him in the ocean. “The SPFA?”

Hogarth  nodded, “Yep, the Society for the Prevention of Furniture Abuse. Their motto is ‘Desks Have Feelings Too’.”

“Go away Hogarth, I’m busy.”

Hogarth broke a branch off the nearest office plant and plopped into a chair next to my desk. “Pretty sure, destroy desk with forehead isn’t the most important thing on your backlog. What’s up?”

I sighed, he wasn’t going to go away so I might as well get this all over with. “The team didn’t take into account planned vacations and the product owner used their expected velocity to promise a feature to sales. Half the team was out and the other half was almost totally consumed by blocker bugs.”

Hogarth asked, “Didn’t they have this issue the last three months ago?” 

I nodded.

“And I seem to recall it was raised after the first sprint, over six months ago. Right?”

I nodded again.

“Well,” Hogarth said. “Sounds just like that problem you had with the original login server not being able to scale beyond 1000 simultaneous logins.”

“Don’t be silly, Hogarth”  I snapped. “You know very well we already fixed that issue two sprints ago.”

Hogarth was smiling, I hate it when he smiles like that, it usually means I’m about to be setup. “That’s right, we did didn’t we? How is it we took care of that?”

I didn’t know where he was taking this and I couldn’t just leave it alone, “We identified it as Technical Debt. We put it into the product backlog and worked with the product owner to prioritize it alongside new feature work.”

Hogarth nodded, “So we had  a problem with the product architecture, we recognized it and put it into a backlog with everything else to be prioritized?”

“Yes!” This was starting to get annoying since he was just reviewing what we all knew. Where was he going with this?

“So where is the backlog for the problems with the team? ” Hogarth asked.

Backlog for the team’s problems, like we do for Technical Debt?  Team Debt?

And he’d done it again…

 

Team Debt is just as destructive as Technical Debt

If you have any doubt that we are worried about the effects of Technical Debt, just Google the term. I get just over one million hits when I do. With the concept of agile development now considered in the mainstream, for software development if not all product development, the recognition that we can’t build our way out of technical problems with new features is taking firm hold. While how we approach it may still be an area of wide debate (TDD, BDD, dedicated legacy teams, rip and replace architecture, etc.) we are coming to a fundamental agreement that we need to slow down and deal with the problems we’ve created before we can speed up and build the next wave.

What about our teams though? It can be very easy to forget that agile is not just a new product lifecycle process (PLC). We have been so ingrained that PLC process is about what we are building that we often forget that agile is as much about “who” is building something as “what” we are building.

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done” –Principle 5  of the Agile Manifesto

Team Debt is the term I use to describe the impediments, issues and blockers that prevent the team from improving. Where testing and product usage will generally uncover your products Technical Debt, it is the Team Retrospectives, manager one-on-ones and coaching observations that will uncover the Team Debt. On the whole, something we are getting pretty good at doing in most agile organizations. Thanks to Scrum, we have built in the concept of the retrospective at the end of every sprint which gives us the mechanism to uncover our impediments.

Of course knowing the problem is only half the battle. We see this with Technical Debt all the time. Sure we know that our infrastructure can’t handle over a 1000 simultaneous logins, or that our core server has legacy code no one even remembers who wrote. We’ve known this for years and we’ve never done anything about it because we were too busy build the next new, wizbang, feature instead.

So just as we need to put Technical Debt into our product backlog and actually prioritize it to be fixed, we need to discover, track and address Team Debt.

The Impediment Backlog

Here, at AOL (Yes, AOL, they  are still around and doing cool stuff in 2016),  I’ve been experimenting with the concept of creating an Impediment Backlog. Just as the product has a backlog of everything we want to do with it, we need a backlog to track the issues impacting the teams. I’ve started coaching my teams to create an impediment backlog and include items from it into their Sprint Planning right alongside items from the Product Backlog. I’m also advocating for the creation of an Organization Impediment Backlog where we track cross and multi-team issues that cannot be solved at the team level.

By making the issues visible and putting them into the same formats as our Product Backlog, we can more easily understand and fit them into our day to day processes. If it looks like a duck, and we’re in the business of making ducks, then it will fit right in. If it looks like a fork and we’re in the business of making ducks then we’re likely to ignore it.

Ask me in a few months how we’re doing, it’s an experiment in the making.