Metrics: Third Rail of Agile Adoption

“Am I good or what?” The question was, of course, rhetorical, I was alone in my office. I

Photo Courtesy of: https://www.flickr.com/photos/bike/

Photo Courtesy of: https://www.flickr.com/photos/bike/

couldn’t help it though, I was pleased as punch and nothing was going to ruin my great mood.

“Or what?…”

Not even an 800-pound invisible gorilla.

“Go away, Hogarth, you can’t ruin my mood today. I’m on top of the world.”

Ignoring me, Hogarth ambled into my office. Spotting my new fichus, he plopped himself down and tore a branch from the tree. Around a mouthful of leaves, he asked: “So, agile adoption going well?”

His question instantly dispelled my annoyance at his assault on my plant. “Yes, yes it is.” I turned the monitor around so he could see. “Just look at these velocity trends! Every team is hitting or exceeding their velocity targets and we’re only three sprints in. It’s absolutely fantastic!”

Hogarth leaned in and intently studied the flat screen display. “Impressive. That’s got to be one of the fastest velocity growths I’ve ever seen. What did you do differently this time?”

“Hey, I’m just good. Awesome training, great coaching. Oh, and I bet the incentive program really helped out.”

Hogarth cocked his head to the side, “Incentive program?”

I leaned back smugly, “Oh, yeah. If the teams hit the velocity goals then they get a cash bonus and we’re going to hold a huge party at Humdingers Resort.”

Hogarth nodded, making some appreciative sounding grunts. I’d finally gotten to him. He was speechless.

After a long gaze at the monitor, he turned to me and gave one of his brilliant white smiles. “Sounds like a real Goodhart moment.”

It was my turn to cock my head in confusion. “A Goodhart moment?”

My gorilla nodded at me, “Uh huh. A well known British economist has an economic concept named after him. The layman’s version of Goodhart’s Law states, ‘When a measure becomes a target, it ceases to be a good measure.'”

I looked at Hogarth. I looked at the velocity reports. I looked at Hogarth. “You mean…”

He nodded, “Yup, those velocity metrics are about as useful as wheels on a speed boat. They look good spinning, but they are not really doing anything.”

 

Metrics- Can’t live with them, can’t live without them.

“Metrics are not bad, managers using metrics improperly is bad.” was a quote I sent out on Twitter (@JBC_GC) during Agile Open Northwest 2017. The session was “Why do metrics get a bad rap?” and it was a lively conversation with some surprising outcomes.

I’ve had a lot of success with the healthy use of team metrics. I’ve used the afore mentioned Goodhart’s Law as a conversation starter on the good use of metrics many times. And despite my success with team metrics I had never really articulated, to myself or others, what the disconnect between team metrics and management’s desire to set goals on those team metrics was. Turns out it is rather a simple thing.

Managers don’t actually care about metrics. They care about success.

Managers want to know if the product will ship on time or with the promised features or with the promised value, or all of the above. They end up using team metrics because it’s all they have. And in the classic square peg in a round hole scenario, they were hitting the square peg with the hammer to make it fit.

And how’s that worked out for us? Even in the well run agile project, the ability to tie team estimates and metrics to actual shipping dates is a highly mixed bag. We are horrible at estimating, to the point that the #NoEstimate movement is gaining traction through being right. This, however, is not a rant about how bad we are at estimating, using no estimates or even bad metrics. It is instead a look at what we can do about this disconnect between Team Metrics and Management need.

Step 1: Recognize managers need better forecasts, not better metrics.

Step 2: Stop using Team Metrics for forecasting.

Step 3: Give managers the tools to let them forecast.

 

Step 1: Recognize managers need better forecasts, not better metrics.

The best use of metrics is “as a lagging indicator of if we might want to talk to the team and see if they need help.” This is the coaching advice I give to management and teams, usually shortly after quoting Goodhart’s law. Metrics will not tell you when a team will get done without the risk that the metric will run afoul of the wisdom of Mr. Goodhart.

Just as you want to know if you should pack an umbrella tomorrow (or wear sunglasses, or get your snow blower ready), management wants to know if the project will be successful. A totally and perfectly valid request. Only you don’t use team metrics for this. Even when metrics are used as the source data, they are still being used to forecast. We look at velocity trends to estimate when a team will be done. That’s forecasting, not metrics.

Step 2: Stop using Team Metrics for forecasting:

Performance based on incentives doesn’t work. When we try and use metrics to drive performance we get Goodhart’s Law. The example I like to give comes from the book Freakanomics.

To summarize: India was having a problem with cobras. To address the problem the government came up with a great idea. They would give a bounty on every cobra head turned into the government. The program was a rousing success, just not in the way the government intended. People started raising cobras just so they could collect the bounty. In short order, India’s cobra problem was even larger than before the bounty was put in place.

If you tell a team they must make a March 15th release date. They will very likely hit that date or come really darn close. And then the company ends up dealing with bugs for the next five years. “When a measure becomes a target, the measure is no longer valid.” If you’re struggling to get your message heard, you might try showing them Dan Pink’s RSA Animate video to reinforce that knowledge skill teams don’t work best from incentives, they work best from Autonomy, Mastery, and Purpose.

Of course, not everyone is going to listen, even when they pay us for our advice.  Which is why I advocate limiting the metrics used and make them interlocking. Game one metric and the others will react in the negative. My four metrics are based on Jeff Sutherland’s three recommended metrics and a fourth I’ve had a lot of success with. They are:

  1. Cycle Time
  2. Escaped Defects
  3. Happiness Metrics
  4. Planned to Done Ratio

If you game cycle time so it’s really short, quality will almost certainly suffer. Let quality slip and you see an increase in cycle time and escaped defects. If planned to done is too high, then quality is probably suffering. And if metric 1, 2 and 4 are all really great and happiness is suffering then you have a strong indicator that you’re burning out your team.

Step 3: Give managers the tools to let them forecast.

This one is really easy. Stop reading my blog and go to FocusedObjectives.com. Troy Magennis has crossed the streams of mathematics, classic project management forecasting and agile to come up with a tool that allows managers to forecast when a team(s) or feature(s) will either be done or if it will hit the desired schedule.

 

I’ve seen this tool in action and heard stories from several users of Troy’s Forecasting approach. With this tool, the managers no longer need detailed metrics from the team. They can build forecasts based on as little as a half dozen data points, which can even be made up for initial forecasts. With just cycle time data points, you can run massive Monte Carlo simulations to get 80-90% accuracy on your forecasts (note, anyone who claims 100% accuracy is also gaming the system, or in denial).

 

So keep team metrics focused on improving the team. Give management their own tool for forecasting schedules and/or capacity. Then watch as the results of this is the teams getting, even more, work done, with greater predictability, and greater happiness.

One Size fits all Gorilla? No more Telecommuting?

“Look, Eric, that’s just not the way we work here.” I was trying to go for my most fatherly voice with this. It was a touchy subject. “You’re an incredible engineer, you’re awesome with the rest of the team. And the mentoring you do is priceless.” I clasped my hands together and tapped the desk with them, “but at this company we just can’t have people marching to their own drummer. You need to adjust your work schedule. We have policies and procedures that apply to everyone and that means you need to be here in the office everyday, or you can’t work on the team…”

 

An hour later I leaned back in my chair with an exhausted sigh. Wow, I was wiped out. That kind of meeting never goes well. But it had to be done, policies are policies and if it’s good for the boss, it’s good for everyone. What’s that saying about “what’s good for the goose is good for the gander”?

 

My office door opened just then. I looked up, half expecting to see a tearful Eric back to plead his case. Instead the light beyond the door was all but blotted out by the broad shoulders of my personal gorilla, Hogarth.


“What do you want, Hogarth?” I wasn’t going to let him get to me this time. I’d been enforcing company policy and I’d done it nicely and professionally. He couldn’t get to me this time.

 

Hogarth ambled in, swinging forward on one arm as the other reached out towards me with something.  “I just came to make a delivery,” Hogarth said. “Here’s your T-Shirt for the company picnic.”  He handed me over a folded black shirt that looked  minuscule in his hands. When I took it from his gorilla-sized hands I found the shirt didn’t look all that much larger in my hands.

 

Holding it to me I glared at my gorilla, “Hogarth, this shirt wouldn’t fit a pygmy. I don’t think my shoulders will even fit through the arm holes.”


Hogarth gave a shrug, “What can I say, Marketing ordered a ‘one size fits most’ people. Guess you’ll just have to adjust yourself to fit…”

 

Ouch, hoisted on my own words. I really have to stop doing that.

 

 

GET TO YOUR DESK BEFORE THE TARDY BELL RINGS!

 

Yahoo CEO tells remote workers to report to the office (Huffington Post)

The recent news, that Yahoo’s CEO has ended telecommuting has started up a firestorm of fresh debate and exposed the coals of long smoldering arguments on the subject.  According to the articles, all telecommuting is ending. From the customer service rep, who works full time from home, to the engineer who works from home one day a week so he can save on that hundred mile commute.

 

The reasons CEO Mayer gives should sound familiar to agilists everywhere:

“Some of the best decisions and insights come from hallway and cafeteria discussions, meeting new people, and impromptu team meetings. “

 

Sounds almost like Mrs. Mayer was reading out of the Agile Manifesto when she wrote her memo.

“The most efficient and effective method of  conveying information to and within a development  team is face-to-face conversation.”

 

Business people and developers must work together daily throughout the project.”

 

I’ve worked on both remote and local teams and I can’t argue with Mrs. Mayer’s general logic one bit. There is nothing that beats face to face conversation. All the best that technology can offer still only scratches the surface of replicating that face to face experience.

 

And yet I find myself agreeing with those that are up in arms about this policy.

 

1: Are we in prison? First and foremost it is a mandatory policy, no or very few exceptions will be given. This reminds me of the agile principle that sits right between the two Mrs. Mayer seems to be playing from.

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

 

Give them the tools and the trust to get the job done. Yahoo’s move isn’t giving people the tools and trust. Instead its making a mandate that covers all teams, all projects, all people, all locations. Even universities and public K-12 education have realized that some students will thrive better with virtual learning, than with face to face. You don’t motivate teams by locking them in a room and feeding them pizza. At the end of the day, the room is still locked.

 

2: Everyone must work in the office, which office? In the Yahoo memo Mayer says ” From Sunnyvale to Santa Monica, Bangalore to Beijing…” Well let’s see here. Say I work in Customer Support and I live in Santa Monica. The head of Customer Support and the largest team are in Bangalore, in fact there are no other CS folks in Santa Monica. Does this mean I have to move to Bangalore? My job is very internally focused on Customer Support. Sure, there are advantages to being able to go to lunch with HR. At the end of the day though, I still am not seeing my team face to face.

 

We live in a global society now. With projects often having dozens if not hundreds of people involved, it is nearly impossible for everyone on the project to be co-located. Someone is always going to be working remote. What’s the difference between the remote engineer sitting in a sales office in Des Moines and a remote engineer sitting in his home office in Windsor Heights (a suburb of Des Moines)? In High Tech, it has become increasingly common for test organizations to be located in India or China. Is Yahoo going to relocate all these people to Sunnyvale so that they can be next to the engineers making the product they test?

 

3: Get the right people on the bus: I reviewed Jim Collins book, Good to Great, back in November of 2011. Mr. Collins makes a very compelling argument for getting the right people on your team, instead of getting the right skills on your team. The same concepts hold true in military special forces, where a team trains together long term and learns the skills they need for a new mission together, instead of bringing a new person in.

 

Say you find the most brilliant engineer for your project. Not only does she have all the skills you need, she fits great with your company culture and the team. You know she’d be a great asset. Only your offices are in San Francisco and New York. The engineer is in North Carolina and can’t move because she takes care of her ailing mother. Do you pass up on the engineer who gets a perfect 10 and instead hire the local engineer who gets a 6, just because he’s local?

 

 

Working co-located is great! – Don’t get me wrong. When I’ve had the luck of working with a team, all in one place, the effects are awesome. The issue here is like an army handing out size ten boots for everyone, because that’s the most common boot size needed. What happens to the guy who wears size 13, or the woman in the size 4s?

 

You do what is right for the team. You remember that your teams are not the entire company. You remember that one size never fits all.

 

Oh! – In small defense to Mrs. Mayer. For all of you remote workers who don’t want to use a webcam in your meetings?

 

Get over it. Working remote doesn’t mean you get to come to work in your pajamas. It means you have to work 200% harder to connect with your other co-workers. Video chat is here, it is real, it is now. Use it, or get your butt into the office.

 

One size won’t fit the Gorilla and it won’t fit your employees either. 

 

Hogarth and the Gorilla Talker