Having goals is a great thing. If there are no goals, how do you know where you’re going? And if you have a goal, how do you know you’re making progress towards that goal?

One of the tools we’ve had in our toolbox for a long time is the use of OKRs — or Objective/Key Results. The “Objective” is the place we want to be. The “Key Result” is how we quantitatively measure progress against that objective. The “quantitative” part of that is very important — hard numbers are exactly what they are, while “qualitative” measurements are a lot harder to track.

knowing how to count is important

For example, a common objective in previous engagements has been “get the developers better at TAS”. Wonderful objective, if TAS is your target platform, we’d expect developers to get better at it through the course of the engagement. The problem with this objective is quantitatively measuring that: “better” is a soft quality.

You can try a self-ranking, but that just leads to “Hey Bob. Go ahead and mark yourself proficient so we look good in front of your boss. If she’s not happy at the end of this engagement we don’t get the team lunch”.

There are proxy measures — how many developers pushed an app? Bound to a service? Used environment variables in a manifest? None of these directly measure a developer’s confidence in the tool set and not all apps are pushed equally. “Hello World?” not a problem. Some react/go combo thing with storage requirements? That’s going to flex some serious TAS muscles.

You could try certifications, but those cost money. And study time. And test taking time. And there’s the (fairly loud) set of opinions that certifications just measure test taking ability, and not actual proficiency. Finding a quantitative measure on a qualitative objective is a difficult thing to do and still avoid gaming the metric.

A note on gaming: ever since metrics have been around, people have been gaming them, especially when things like financial compensation are tied to those metrics. For example, a company rewards developers by measuring productivity in terms of lines of code written per hour. That just encourages developers to copypaste as much ugly, repetitive code as they can to give the appearance of productivity. Ok then, how about tying it to story points? Developers would just pad their points under the guise of “uncertainty” to keep that cash flow coming. Measuring number of bugs found and fixed? I put the over/under on QA teaming up with the devs to conspire to get the bug money train rolling at 2.5 hours. *Any* metric tied to compensation will eventually be gamed at the expense of customer satisfaction.

credit Scott Adams
credit Scott Adams

So now we have two problems: find a way to quantitatively measure progress against a goal, and find quantitative measures where gaming them does not provide any ancillary benefit.

Let’s take an example: In a previous post, we talked about the practice quality of “having a place to experiment”. A trivial KR for that is: “is there a TAS instance that any developer can log into at any time?” It’s a yes/no KR, and yes, we have one. And it has a fairly broad marketplace of services, so there’s a lot a developer can do there.

But what’s a more meaningful measure there? A platform is great, but if no one is using it then maybe the platform experience isn’t as great as we think. Or maybe its harder to get to than we think. Or we didn’t market it well enough. Or something else. So we should be measuring something like “number of apps pushed to the platform, sorted by unique developers”. This is a lot more useful — it tells us that the availability exists, and that the platform is being used. Further, devs pushing to this platform doesn’t change developer compensation at all: the only motivation to push there is to try things out and improve TAS-related skills. The likelihood of gaming this is very low.

Next step: Can we even measure this? Ultimately, yes, there exists an API we can query to see regular activity on a TAS platform, and tools exist to make this easier that can generate regular reports on this activity. This appears to check all the boxes: it’s quantitative, it’s meaningful, it’s not likely to be gamed, and it can be implemented without a large effort.

As a leadership team we’re working to decide on good KRs and get them implemented to start tracking progress. We’ll keep the team up to date on this progress.

Practice Lead with VMware. Outdoor enthusiast. Amateur banjo player.