Kristan Wheaton Assistant Professor Intelligence Studies Department Mercyhurst College

One of the more interesting theses I have supervised over the last several years was Lindsey Jakubchak's The Effectiveness Of
Multi-Criteria Intelligence Matrices In Intelligence Analysis. Lindsey's thought was to take a version of the well-tested school of operational methodologies often referred to as multi-criteria decision-making methods (MCDM) and flip it on its head to turn it into an intelligence method. The results of her experiment show the method as promising in a number of different respects, though, clearly, there is still work to be done.

For those of you unfamiliar with MCDMs in general, there are many, many variants of the process and each is accompanied with all of the arguments and counter arguments typically associated with academe. Lindsey just wanted to see if there was any value in her proposition at all, so she chose one of the simplest and most common forms of MCDM to "flip"—a streamlined version of the US Army's Staff Study Method.

What do I mean by "flip?” MCDMs are typically used to help select the most logical course of action based on a given set of criteria. Say, for example, you were looking to buy a car. You had down selected to three particular SUVs but you couldn't make up your mind which one was best for your family. An MCDM would ask you to select the criteria you thought were important to you and your family (seating, reliability, gas mileage, storage, etc) and then rate each car, using a matrix to sort the results. Arguably, the car that best meets your criteria is the one you should select (Anyone familiar with Consumer Reports, for example, knows that this is the way they come to their conclusions about various products).

What if it is not you buying the car, though? What if you are trying to figure out what kind of car a friend might buy? Your friend might prefer sports cars to SUVs and have an entirely different set of criteria for choosing one. That's what I mean by "flip.” What if you could use an MCDM not as a tool to help you make better decisions but as an intelligence analysis method to help you figure out what an enemy, a criminal or a competitor is likely to do? That was what Lindsey set out to test and she gave her method a name - the Multi-Criteria Intelligence Matrix.

While, due to the topic she decided to explore—Russia's relationship with OPEC—she was not able to evaluate forecasting accuracy (though I give her full points for trying), she was able to compare her experimental group to the control group in a number of other interesting ways. Using the standards in ICD 203 as a guideline, she was able to say a couple of interesting things like:

"Although the experimental group indicated a lower level of knowledge in regards to the topic (Russia’s relationship to OPEC) and expressed a lower level of interest with the topic, both of which were found to be statistically significant, the experimental group was able to arrive at a broader range of possible Courses Of Action (COAs)."

"The average completion time for the control group was 70 minutes and the average time for the experimental group was 58.6 minutes. Therefore, when looking at the big picture, although the experimental group seemed less knowledgeable and less interested, they were able to arrive at a more complete list of relevant possible COAs, and they completed their analysis in less time."

"While a few students in the control group provided one or two alternative COAs, the majority of the student-analysts merely provided one COA with few comparisons to any alternatives, thus not providing any insight to whether or not alternative solutions were considered. In the experimental group, the student-analysts, who used MCIM, provided a list of all possible COAs, and identified the importance of specific criterion or various factors to those COAs."

In the end, the study suggests the method has promise and, with Lindsey's results in hand, it has more evidence to back it than many other, more widely taught, methods. Download the full thesis here.

Download the full thesis here.

Reference:

The effectiveness of multi-criteria intelligence matrices in intelligence analysis. Lindsey N. Jakubchak. A thesis submitted to the faculty of Mercyhurst College in partial fulfillment of the requirements for the degree of Master of Science in Applied Intelligence. May 2009. 131 pages.

About the author: