The objective of the FOCUS program was to enable "counterfactual reasoning," defined as "statements about what would have happened if different circumstances had occurred. For example, a postmortem review of an analysis failure may lead to a conclusion that analysts would have avoided the failure if they employed better analytic tradecraft; perhaps by having double checked assumptions, perhaps by having considered a broader range of hypotheses, etc. Counterfactual forecasts about what would have worked in past circumstances are very often the basis for lessons learned for what to do in the future. And such lessons often evolve, over time, into best practices and tradecraft."

But how can we test reasoning about intelligence failures? What about the U.S. evacuation from Afghanistan? Right now, there is a rush to assign blame on the Biden administration by Republican politicians, and to cast blame on past Republican Presidents by Democrat politicians. Who is right? The problem is that we can't rerun history to find out whether President Trump, if he had been reelected, would have done a better job.

However, in the FOCUS program we did rerun history -- that is, histories of bots competing against bots in computer games. Then we analyzed how well we counterfactual forecasters (including BestWorld's Dawna Coutant and Carolyn Meinel) were at predicting what would happen in these alternate histories by comparing our counterfactual reasoning with what actually happened when those evil geniuses who created these bots at the John Hopkins Advanced Physics Lab reran history.

Wow! Thanks to the training team at Good Judgment 2.0, led by the same people who inspired us to launch BestWorld, we did really well! Next, IARPA hired some Amazon Mechanical Turks -- people who make a living participating in psychology research projects -- and our training team taught them, too, how to be super good at counterfactual reasoning.