Metrics Done Wrong

Posted on June 28, 2012


CC-by-SA  short link:

When talking to agilistas, several times I heard the following argument, in several manifestations, against the collection of software metrics, and internal metrics in particular.

“Do not use internal metrics because people tend to use them to measure individual programmers’ performance. This defeats the purpose of team cohesion, fomenting individual-centric performance over the team or group.”

Define Your Terms

As Voltaire would say, “define your terms”. So let’s get started. In this conversation, internal metrics would be those for internal use (internal to the development team or department.) External metrics would be those to be consumed by stakeholders external to the development process (clients, shareholders, upper management.)

With that out-of-the-way, this argument does not hold any water. It is a bullet-ridden bucket.

In my opinion, this is not a reason for avoiding  internal metrics. What the example counter argument above tries to describe is not a problem with metrics at all. Instead,  it is a procedural, educational and behavioral problem within the organization.

Tools and Wielders

That is, such an organization is using a tool for the wrong purposes. Worse still, it is a wrong purpose that is severely detrimental to the organization’s bottom line. It is not a problem with the tool, but with wielder.

Or as Thulsa Doom of Conan’s lore would say, “What is steel compared to the hand that wields it?”.

It is a case of using the wrong tool for the wrong job, or worse, using a good tool inappropriately.

Just because you can…

Bad Arguments Are Typically Isomorphic

Such an argument is  isomorphic to one made against source code comments (sadly I’ve had to deal with them.)

“We don’t use comments because when the code changes, comments become obsolete.”

And the world wept in horror. Duh! When code changes, so must the comments. That is, valid, up-to-date comments are part of the deliverable. Hammer for nails. Screw drivers for screws, and so on and so on. A tool is built for a set of closely related problems and purposes in mind.

Same with metrics in general, and internal metrics in particular. People using metrics the wrong way and/or for the wrong purposes is not a problem with the metrics.  It has no bearing whatsoever in the correctness of a tool.

It Is a People’s Problem

Fundamentally, it is a problem with people. Blame it on education or politics if you will. It changes nothing. The nature of the problem is of no consequence (beyond the need of corrective actions) to the true nature and intent behind internal metrics.

If an organization does not use internal metrics because THEIR people use it to measure individual performance throughput, then the organization has more severe things to worry about than metrics alone.

Also, and here I am somewhat in disagreement with Martin Fowler’s well-known position on the subject; measuring individual performance does have a place.

To Manage, You Must Measure (Somehow…)

For example, if you hire a senior software engineer, you expect his performance to be greater than one from a junior developer. However vacuous and nebulous your impression of acceptable performance is, this measure is apparent, and measuring it is reasonable.

You can’t describe it, but you know it when you see it.

Would you be so foolish, in the name of team cohesion, to avoid measuring performance (or lack thereof) in such a situation, however tacitly?

Here is another example: You need to track the rate of bugs introduced per feature or SLOC by teams and individuals. If a team or individual has a substantially smaller rate than the average, then the company can learn from them, learn what they are doing right and attempt other teams and individuals to leverage such knowledge.

If, OTH, a team or individual is sustaining an unusually high bug/feature rate, then the organization can investigate the root cause. Is it the source base? Tools? A skill deficiency that can be remedy in-house? Can the situation be salvaged, or should the team/person be let go?

These are valid questions any organization must be able to answer in order to operate effectively.

Moreover, there is nothing intrinsically wrong, evil or impractical in measuring individual performance. There can be cases when such a tool is used to look for scapegoats, but that is a different issue. Somehow in software, and in the agile movement in particular, certain individuals build these belief cults that are ultimately orthogonal to the business at hand – the business of creating software for someone who is paying for it. The business of creating something of value with acceptable costs and ROI.

The Meaning of Agility

That is not what agility is about. Agility is not a cult or an intrinsic moral system. It is a method to conduct engineering business, and software engineering/development business in particular. In any business, measuring individual performance is necessary.

Fool the one who does not heed this truth.

You can use a hammer to build something, or you can use it to smash someone’s head. Is it the hammer that is faulty?  Is the hammer an intrinsically wrong tool?

No really, just because you can, that doesn’t mean you should.