Metrics to Measure Mission Success




I am retired now, but as a systems engineering on intercontinental ballistic missiles, a typical concern of mine (and many others) was to continually look at large amounts of data to draw conclusions on the readiness of the weapon system to meet its mission. The mission of intercontinental ballistic missiles is to deter war through readiness, and if deterrence fails, to take out the enemy’s military capability. 

I needed to understand, is the reliability improving or decreasing? Is accuracy remaining adequate? Will the missile still fulfill its mission under the effects of nuclear war? Fundamentally, were the readiness metrics trying to tell me something?

An important corollary question was always: “Is this a real trend, or is there something wrong with the data or how I am looking at it?” Strict systems engineering disciplines and strong leadership made this process very effective.  It can work quite well when done correctly. 

In my situation, the mission was clear and constantly discussed. Analysts, especially the contractor, were incentivized for honesty, precision, and persistence. I was privileged to work within a metrics system that worked and worked very well.

How do you find the right metric?

The best general definition of metrics I have found has been from the Air War College lesson on “Quality in Daily Operations”, circa June 1995 textbook: 
“When you’re developing metrics, you need positive, open communication with customers, suppliers, leadership teams and process experts. To help you get started on the right path, here are eight characteristics of a good metric:
  • Meaningful to the customer
  • Simple, understandable, logical and repeatable
  • Shows a trend
  • Clearly defined
  • Data that’s economical to collect
  • Timely
  • Drives appropriate action
  • Tells how organizational goals and objectives are being met through processes and tasks”
Unfortunately, there are too many examples of top leaders who fail to seek out good metrics that reflect their mission. Rather, they seem to seek out metrics that are simply manageable and meet certain “sub-optimal goals”, such as nice-looking annual reports or quick sound bites.  

The classic case of poor use of metrics is “body counts” during the Viet Nam War. The 3 television networks were happy to report numbers of enemy people killed versus numbers of our people killed on a nightly basis (at dinner time). It was an easy metric to show to millions with little chance of misunderstanding. It serviced the government’s desire to show what looked like good news. It seemed to indicate always that victory was just around the corner.

But metrics must be closely coupled to the mission. With the mission in Viet Nam not entirely clear, this was always problematic. Was the goal to stem the tide of Communism? Reunite North and South under a South that was nominally “democratic” and capitalistic? Leave with honor? Do we even know today what the primary mission was back then?

What we got from the Vietnam War metrics was the dull drumbeat of fatalities in a mission that was hard for most nightly news watchers to comprehend.  Actually, if the goal was to end the war no matter the consequences, perhaps body count was the perfect metric.

Good leaders look for the real goals and create real metrics. If the mission is unclear from the outset, that road is steeply uphill. But even with a clear goal and good metrics, there is another trap as exemplified by the current VA wait times scandal.

What if you cannot trust your data because you have somehow motivated your middle managers to “fudge” the data? 

The best top leaders know how to anticipate the problem of bad data entry and create a system that avoids this, or at least detects it. The top leaders at the VA certainly had previous IG reports to give them strong hints of what was happening. And how could they not be interacting with their ultimate customers, the veterans, for anecdotal data?


So the moral of this story is, whatever you are in charge of, spend the time necessary to create, operate, and update your metrics and your analysis processes. This may even push you to clarify and better publicize your mission, which is always a good thing.  Although the payoff is in the long run, it is well worth the time.

Comments

Post a Comment

Popular posts from this blog

Improve Your Position

Image
In 1980, getting to the point of takeoff with this 25 year old aircraft was a real maintenance challenge We are still flying these aircraft, albeit the water-injected engines have been replaced A KC-135 in-flight refueling mission often started with my radio alarm going off at 3am. Aware of my wife's low and steady breathing, I quickly kill the alarm and quietly rise from bed to take a shower. I had set the alarm for 3 because I need to be at the air refueling rendezvous point on time. There is a precise spot out there in time and space. If I am not there, the receiver does not get his fuel. Recon does not happen. Top military and civilian leaders will be without the timely intel they need to keep the peace in the Cold War.  So that's my mission. Get the receivers the fuel they need to do their mission. Not really all that glamorous. But important. Now fully awake from the bracing shower, I tiptoe around the bedroom to get my flight suit and other gear my as wife sleeps p...
Read more

My 1st USAF Operational Assignment, Inflight Refueling the SR-71

Image
Flying KC-135s was my first assignment after graduating from the Air Force Academy and after general pilot training. I was co-pilot and then aircraft commander. Born in 1952, my life was very much affected by the years immediately following WWII. This was a time of optimism, a tremendous boom in aerospace, and dreams of a push-button future. My Dad’s B-24 bomber was pretty amazing technology that helped win the War. But post-war, the B-24s were scrap and there were now screaming jet engines, electronic computers getting smaller all the time, and advanced aircraft that could fly to the edge of space itself -- the stratosphere. One of these aircraft was the KC-135 Stratotanker whose operational ceiling was 50,000 feet. Above that, the pilot wears a spacesuit.   The first KC-135A rolled off the assembly line in 1956. They remain key assets today. I flew them from 1977 to 1982. The KC-135 aircraft is a Boeing product and looks a great deal like the Boeing 707, which was its contemporar...
Read more

My Crew

I was a cautious, safety-minded aircraft commander when I flew KC-135 tankers. I read all the pilot bulletins and Strategic Air Command safety magazines, so I knew how a moment of inattentiveness could lead to tragedy. And if my aircraft went down, it would not just kill me, but also my crew, others on board if we were hauling passengers, maybe the aircraft we were refueling, possibly people on the ground. I combated inattentiveness with a few tricks including good habits. If you do it right each time, no matter what, then when you are in a hurry you will do it right. For instance, the old tanker was a “water wagon”. It needed de-ionized water injected into the engines at take-off to get off the ground. During the aircraft commander’s walk around, one of the things we looked at was a fan recessed into the engine cowling. I would give that fan a twist at each engine to make sure it would turn. Sometimes, in very cold weather, that fan could freeze and that would mean no water and no ext...
Read more