Kudos to OMB and Federal agencies for completing the ratings of 100% of their major IT investments, which by the way was no small undertaking .  So we now have a complete and well presented view via the Federal IT Dashboard (http://it.usaspending.gov/) of how well our “major” Federal IT investments are adhering to cost, schedule and various risk (via the Evaluation Factors) metrics.  Assuming the data is and remains accurate and current (and sure, why not , we have a pretty good picture of the Investment side of the ROI equation.  

As OMB seeks to continually improve reporting via the IT Dashboard and facilitate effective management of our Federal IT systems, I’m hopeful that OMB’s next step will be to focus on the Return side of the equation.  After all, we don’t create and manage projects simply to have them completed on time and budget.  While some investments are still under initial development and thus have not yet had the opportunity to produce a return, many of the “majors” are in the Operations and Maintenance phase where they should be delivering value (return) to their end users.    

Measuring return, especially in the Federal government where not all benefits can be monetized, can be difficult, but there are ways that this can be achieved.  For starters, a simple measure of customer satisfaction might be an informative addition to the Federal IT Dashboard.  

Eileen Federic wrote an interesting article in Baseline which addresses “getting the right information” for a decision. There are many important ideas in this article, and no one will argue that getting the ‘right’ information is necessary, but it is not necessarily the ‘key’ to making the best decision. In today’s world, we often have more information than we can process effectively, especially when it comes to ‘crucial’ decisions that can result in the success or failure of a project or business.

Just as important as having ‘good’ information, and sometimes more important, is the ability to synthesize the information to make a decision. This is no easy task, given that all important decisions have multiple, conflicting objectives, and most important decisions have multiple decision makers who, if left to their proverbial ‘gut feel’, as you put it, would not necessarily come to the same conclusion. Hence, arguments, hurt feelings and loss of time are too common.

The decision on how much time and money to allocate to gathering information is in itself an important decision. But no matter how much time and money are allocated for this purpose, there will always be some missing or ‘untrusted’ information in any important decision. The quality or veracity of information that is at hand, as well as uncertainty and risk due to incomplete information must all be taken into account. Eileen recognizes that there is much more than getting the ‘right’ information, when, at the end of her article, she observes that “Clearly, facts are the foundation of the decision-making process, but facts without an analysis and understanding of consequences aren’t enough to achieve hoped-for results. We need a convergence of accurate information, the right technology and processes to analyze that information and provide business insight”.  However, analysis, (meaning to break things down) is only part of the process. Because all important decisions are multi-objective, synthesizing the parts into a whole is even more of a challenge and needs to be addressed in any effective decision process. This synthesis must include not only factual data; intuition and subjective judgment must play a key role in interpreting the data and synthesizing the information.

There are several decision process that have been developed over the last half century or so that address the difficulties and approaches to making important, multi-objective, decisions. One such process that is both theoretically sound and which has been effectively applied to thousands of decisions is call the Analytic Hierarchy Process (AHP). I have been teaching AHP at The George Washington University School of Business for more than 25 years and it is now being taught at many of the leading Universities in the United States as well as abroad using software I developed with Expert Choice.

Ernest Forman
Professor of Decision Science
The George Washington University