Challenge 2 – Valuing Probabilistic Foresight

Of all of the Three P’s challenges, the devaluing and avoidance of probabilistic foresight work by too many futurists and foresight professionals is our most important challenge, in my view. It is the topic, in various ways, of the first three of our eight challenges.

When our modern field began, with the founding of the first foresight think tanks like SRI and RAND in the late 1940’s, our best work was probability-centric, with preference foresight as a distant second priority. Our field was too focused at the time on data, trends, and formal modelling, so much that it limited our use of the other Three P’s processes. By the 1960s, we the field had become much more Three P’s balanced, with the emergence of foresight think tanks like IFTF and IAF that explicitly addressed all three practice types.

The 1960s and 1970s also gave us leading books, like Kahn and Wiener’s The Year 2000 (1967),  Alvin Toffler’s Future Shock (1970), Jib Fowles Handbook of Futures Research (1978), Olaf Helmer’s Looking Forward (1983), which took a conscious, even masterful approach to all three foresight types. The best of these books challenged foresight practitioners to be researchers, quantitators, and forecasters first, alternative and uncertainty seekers second, and preference mappers and managers third, echoing our order of operations for the types.

As social complexity and speed increased in the 1970s, every profession became more specialized, and a few, like foresight, overspecialized. Our Western futurist and foresight communities are among those that overspecialized, drifting more into possibility foresight. In 1982, J. Scott Armstrong founded the International Institute of Forecasters (IIF), to advance the practice of forecasting. Rather than actively embrace this new community as a leader in the foresight field, the main futurist associations let this and other probability-based work drift increasingly outside our borders. Even our academic foresight training programs began to neglect probabilistic and quantitative thinking, forecasting and prediction.

At the same time, a misunderstanding of the complexity science and chaos theory emerged in the 1970s, and a culture of the “unpredictable future” developed in many of our academic foresight training programs. This culture in turn attracted imaginative and creative people who prefer that way of looking at the future, but it also increasingly alienated the critical quantitative and forecasting-oriented people that we need in order to keep foresight training centered on the Three Ps.

Once a non-quantitative, non-prediction culture emerges in any practitioner community or training program, it can be hard to restore it to balance. The creatives naturally want more creatives to associate with, and they tend to look at those who don’t think like them as “other”, and vice versa, for those in quantitative communities like the IIF. But the Three Ps tell us that great foresight  needs all three types of work to be done, by a large and cognitively diverse crowd.

Given this history, many academically-trained futurists and foresight practitioners today are guilty of undetermination bias, of imagining the future as freer, less predictable, less constrained, than it actually is. We discuss this and other Emotional-Cognitive Biases in Chapter 2. We see so much around us that is unpredictable, it is easy for us to assume that this is the whole of reality. Yet as the 95/5 rule tells us, there are always special subset of things that are statistically predictable. In living systems, and perhaps also in our sociotechnical systems, if they are replicating and adaptive complex systems, seeing and managing that predictable subset is just as important as seeing and managing the unpredictable, if we are to be adaptive.

We can observe hidden predictability at every scale and in every natural environment. For our universe as a system, such processes as thermodynamics, nuclear physics and classical mechanics, which we understand well, and processes like dark energy, accelerating change, and information and intelligence growth, which we don’t yet understand well, all offer us predictable trends and destinations we must learn to better see and manage. In our global, organizational, and personal domains, there is also much that is predictable, as we’ll discuss. Science already tells us that several processes and events in our STEEPS futures are convergent destinations, single, predictable outcomes that human societies everywhere are funneling toward, regardless of our individual choices, or more curiously, as a result of the average distribution of those choices.

Yet even today, too many well-meaning professionals still declare a personal belief that “the future cannot be predicted”, at least in meaningful ways in the area in which they practice. In their current world view, they can see few probable trends and futures that are relevant to the strategy, plans, and actions they consider for themselves and their clients. As a result, they naively imagine the future can be crafted to their whim. Postmodernists are but one example of this, with their belief in a deeply subjective and unscientific world view.

Such individuals miss all the STEEPS forces and constraints that are converging on predictable futures all around them. They have no particular inclination to see or discuss these obvious trends and destinations or share them with their clients, and use to their best advantage. Even when they agree on the probability of something coming, they see no reason why that thing is more important to understand than, say, the latest novelty in the news. Addressing this bias in ourselves is the first challenge we must all face if we are to become better at foresight work.

The World Future Society, our field’s oldest organization, has long subtly championed this lopsided view. For example, the opening lines in the WFS 2014 conference brochure read: “The future is not a destination. It’s the end result of the actions we take today.” So too Reddit Futurology, today the largest online platform dedicated to the future, with 13 million “Futurists”, has the following language in its definition of the field of “Future(s) Studies” on its home page: “practitioners realize there is no single future, only alternative futures ahead.” The home page of the new School for the Future of Innovation in Society at ASU proclaims, in adaptation from Shakespeare’s Julius Ceasar: “It is not in the stars to hold our destiny, but in ourselves.”

As we will see, all of these statements, innocuous as they might seem, are actually hubris, and dangerously biased half-truths. They assume that foresight doesn’t begin with identifying all the processes and futures that seem implicit in complex systems, waiting to predictably emerge. A good foresighter learns to see that both probable and possible futures always lie ahead, and that the preferable futures we seek to create are always subject to both the predictable and the unpredictable futures that will simultaneously occur. If any of these Three P’s perspectives is ignored or minimized, our foresight is greatly weakened as a result.

Even the Association for Professional Futurists, our fields most developed professional organization, claims, in the second sentence of their “What is a Futurist?” page: “It is not the goal of a futurist to predict what will happen in the future.” This perspective is simply wrong. It tries to “define away” a third of our field, simply because the committee that approved it is not yet appropriately valuing probabilistic foresight. It ignores futurists like (the now late) Hans Rosling of Gapminder, who make constant forecasts and predictions in their work and presentations.

Foresight professionals who don’t like to do prediction, perhaps because they don’t feel confident in any particular futures, or perhaps because they prefer not to share their internal confidences, need to stop saying that prediction and uncovering the probable are not key parts of our field. They are. Some of us may wish the world had no predictable futures, but wishing doesn’t make them go away. At a time when startups are launching blockchain- and cryptocurrency-based prediction markets, it’s time for everyone to recognize prediction and probability are foresight fundamentals, as the Three P’s model reminds us. Understanding as many as possible of our highly likely destinations is critical if we are to make wise strategies, plans, and actions in other, less predictable parts of the future that lie directly in the realm of our free moral choice.

Gidley 2017

Some practitioners will grant there are parts of the future that are highly predictable (demography, humanity’s negative impact on the biosphere, information growth, accelerating technological change), but at the same time, they will misconstrue quantum physics, nonlinear science and chaos theory to argue that there is no predictability “where it matters,” in social and organizational issues and questions. That view is simply wrong, as we’ll argue throughout the Guide, and in most detail in Chapters 7 and 11. Even Jennifer Gidley’s well-researched and often excellent introduction to the history and practice of foresight, The Future: A Very Short Introduction (2017), is guilty of this perspective. This brief book says, in its inside jacket, that “our traditional belief in a predetermined future has been challenged by quantum mechanics” and that “we’ve always had multiple possible futures over which we have infinite control”.

Gidley expresses both of these views throughout her book, but both perspectives are dangerously incomplete. With respect to the universe, there are at least two opposing and fundamental physical theories that describe change, sets of theories that I suggest we call evolutionary and developmental. The evolutionary set, which includes quantum mechanics and nonlinear science, describes unpredictable and divergent processes of change. The other is the developmental set, which includes relativity and thermodynamics, and describes predictable and convergent processes of change. Cosmology hasn’t yet learned how to unify quantum mechanics and relativity into a single theory. One of the quests to do so is called the search for a theory of quantum gravity.

Science to date tells us that our universe is creating both unpredictable and predictable futures at the same time. Quantum mechanics has led inevitably to all kinds of general predictability in complex physical and chemical systems throughout our universe, and on Earth and presumably elsewhere, in biology, society, and technology as well. We also know that complexity and intelligence that can be generated by these predictable and the unpredictable processes, and the control we can have over them, are sharply finite. All physical systems are fundamentally incomplete. No one ever had “godlike” omniscience or ability to control our futures, and nothing in our physical or informational universe is “infinite”.

As we’ll see in Chapter 11, if something like the 95/5 rule operates in our universe, then while the vast majority of our future is best thought of as freely chosen evolutionary paths determined by us, a critical minority of our future will always be developmental destinations, determined by our universal environment and its laws and structure, unfolding all around us. It is most reasonable to predict that many special destinations will arrive in roughly the same way on all Earth-like planets in our universe, whether we individually want them to do so or not. These futures are statistically “predestined”, to use a word that many futurists find displeasing, but nevertheless, is clearly part of the story of change, in all complex systems in our universe.

For a few obvious examples of universal predictability, we know our Sun will die, within a specific range of future years, and that we must eventually must go somewhere else, if we are to survive. For a less obvious example, many courageous scholars have observed that top complexity in our universe appears to be “striving” to move sequentially from physics, to chemistry, to life, to minds, to technological minds over time. It is most reasonable, with the science and evidence we have to date, to expect the emergence of technological intelligence to be a single common future across our universe. Several such special universal processes, including accelerating change, seem to be occurring either partly or mostly outside of our control.

For our own futures, we know that biological humans are going to continue to grow old and die. Anyone who isn’t a naive transhumanist can confidently predict that we won’t be able to change that fact in the forseeable future. All higher organisms age (fall apart) over time, in breathtakingly complex ways at the molecular scale, and we still have very poor understanding of and influence over those systems. We also know that our digital information will continue to persist after we have died, for as long as our society deems it to be valuable. Digital technological systems are far better at protecting their information than biology. We can also see and measure the greatly increasing resiliency and redundancy of the knowledge we use to run our modern societies, knowledge that makes them significantly more immune to complexity regression than any previous societies that have existed.

Though many pessimists forget this fact, no single catastrophe we can presently imagine could seriously set back the hard-won, stratospheric gains we’ve made in our scientific and technical knowledge since the Enlightenment, unless that catastrophe was so large that it erased our entire species and its many redundant archives in the process. Just one surviving tribe, with our current libraries of technical knowledge, could bring us back to our current complex societal state within a millennium. On an astronomical scale, such a catastrophe would be a blip in our record of accelerating change. Any current catastrophe would have to be many orders of magnitude higher than past castastrophes (pandemics, supervolcanoes, meteorites, etc.) in order to slow us down as long as, say, the Dark Ages did. For more on that, see Lewis Dartnell’s The Knowledge: How to Rebuild Civilization in the Aftermath of a Cataclysm (2015)

We can forecast an exponentially growing standard of living, at least as long as our tempers hold and our environment continues to survive its mistreatment. For more on that, see Charles Kenny’s 2015: The Best Year In History for the Average Human Being, Atlantic.com (2015), and Kenny’s Getting Better: Why Global Development is Succeeding (2011). My favorite weekly, the Economist, also does an excellent job reporting our exponential advances. For some Economist data on the impressive gains made by the poorest people in the world over the last 25 years, read Warren Buffett’s Best Investment, Bill and Melinda Gates, Gatesnotes.com (2017). Technology-aided global philanthropy, from institutions and from individuals, has become a powerful new force for good, and it’s still in its infancy.

With the connectivity, quantification, and AI that are coming, we can expect digital systems like groupnets to help us manage the deprivations and violence that some of us face daily, and bind us into one global family more tightly than ever before. See my post The Internet of Families: The Powerful and Intimate Future of Groupnets, Eversmarterworld.com (2015) for more. Groupnets and other societal developmental futures are happening because they predictably satisfy some of our deepest biological preferences. Our existing email, social network, chat, image, and video communication networks are today’s best realization of this desire for increasing intimacy, though today’s networks give us shallow connections, with people who don’t deeply share our values.

We can also predict a continued decreasing average intensity of interpersonal violence as our societies and institutions continue to develop, even as we don’t yet fully understand the causes of this global megatrend. See Stephen Pinker’s The Better Angels of Our Nature: Why Violence Has Declined (2012) for some of the models, a data set that is reasonably good for at the last thousand years. Violence reduction and transparency reforms and processes are accelerating all around us, today though you wouldn’t believe it if you read the evening news, which reports every calamity in obsessive and unhelpful detail, rather than spending most of the time reporting the solutions and strategies that are allowing us continue to shrink and better regulate the violence that remains. This reporting bias will change over the next two decades, as we argue in Chapter 8, but our technology will have to get a good deal smarter before that happens.

The Rise of Prediction Platforms

Many of us don’t realize how many aspects of our social, economic, and technological future are now probabilistically predictable. Even the weather, a classic chaotic and nonlinear complex system, has seen great improvements in predictability. Our prediction horizon for useful weather forecasts has grown from two days ahead in the 1950s, to a twenty-one days in our current best weather models, with the use of increasingly high resolution and distributed sensing, deterministic forecasting, ensembles of competing models, and better human judgment.

As the world digitizes, more and more of its systems are being forecast in probabilistic ways on a new class of prediction platforms. For more on this global social megatrend, read Nate Silver’s The Signal and the Noise (2012), Patrick Tucker’s The Naked Future (2014), and Eric Siegel’s Predictive Analytics (2012). Each of these is an introduction to our recent great strides in probabilistic prediction. As Siegel describes in his chapter on IBM’s Watson, the computer that beat the two best humans on the planet at the Jeopardy! quiz game in 2011, increasingly fast computers, big data, and simple statistical and crudely biologically-inspired algorithms (e.g., artificial neural networks) allowed computers to predict the most useful answer to human queries across a vast domain of cultural knowledge. Google’s DeepMind, a deep machine learning system that has many parallels to the way our own brains think, is perhaps the most complex and adaptive example of these prediction platforms.

The Good Judgment Project
(U. Pennsylvania & U.C. Berkeley)

Consider also Philip Tetlock’s IARPA-sponsored Good Judgment Project, a collective forecasting and prediction platform that weights the inputs of a cognitively diverse group of amateur forecasters invited from the general public. Now in its third year, this research platform is outperforming even CIA analysts at many types of social, political and economic prediction as this NPR article, and Tetlock and Gardner’s new book, Superforecasting: The Art and Science of Prediction, 2015, attests.

Tetlock & Gardner 2015

Read Superforecasting if you still have a conscious or unconscious bias against finding more of the probable future for your clients. Tetlock’s book demonstrates the promise of using evidence-based methods in prediction, and it shows the great promise of a still-emerging foresight tool, online forecasting and prediction markets. Prediction markets are finally emerging for each of the most important topical domains of foresight, which can be remembered by the acronym STEEPS (science, technology, economics, environment, politics, and society). Such markets are a great crowd-wisdom tool you can use to explore futures with your clients. For a few examples, Metaculus is a new science and technology prediction market, and PredictIt is a new real-money political prediction market.

Augur is decentralized real-money prediction market built on top of the Ethereum blockchain (a platform similar to Bitcoin). Its leaders seem quite a bit too utopian for this startup to be likely to develop as a serious new platform present, but it is just one of many new blockchain-based efforts at incentivizing and accelerating social prediction, at tapping the “wisdom of the crowd.”

Primer is a new AI cloud platform, just one among scores that have emerged since 2012, that uses machine learning to help analysts interpret masses of online text and stories to produce predictive analytics, uncover trends, and make simple forecasts. Scores of new cloud platforms focus on predictive marketing, and others on predictive intelligence for governments, business, and nonprofit clients.

For examples of businesses on the frontier of probabilistic prediction, see Palantir and Recorded Future, two of a new breed of web intelligence platforms that use an ensemble of social network analysis, semantic analysis, data analytics, and human intelligence techniques to visualize relationships and make forecasts on current and historical data. Just as with weather forecasting, coming versions of these tools will greatly improve our collective and statistical foresight with previously unpredictable systems.

Another new collaborative foresight and strategy platform is Wikistrat, a “massively multiplayer online consultancy (MMOC)” that runs multiday strategy and policy simulations with 100 or so of its 1,000+ online experts in an online version of two fifty year old foresight tools, the Delphi method and Organizational Wargaming. We’ll explore Delphi further in Chapter 4.

One of the most ambitious predictive platforms I’ve seen yet is Rootclaim, which is constructing Bayesian inference models to improve prediction of social, economic, and political events. See this excellent article, Israeli Startup Develops “Ultimate Truth Machine”, Haaretz, 10.15.17, for more on how that process works, and some of its recent predictions. Bayesian inference may be the fundamental way that all intelligent systems make predictions in our universe. Improving crowd Bayesian approaches is therefore a very promising frontier in predictive foresight work.

The “predictioneer” Bruce Bueno de Mesquita is also a Bayesian modeler. See his (unfortunately too egotistical and self-promoting) book, The Predictioneer’s Game (2010), for more on the long history and value of such predictive analysis. Such models may only tell us the average expected future, and be poor at surfacing black swans, also known as wildcards (unlikely but important events), but they are far less used today than they should be, because our culture, as a rule, still ignores and devalues probabilistic foresight in a wide range of complex systems.

Consider also the European Commission’s Futurium digital futures project which engaged 3500 people in online and in-person workshops over three years, eliciting visions for a digital Europe in 2050. This is an inspiring example of collective preference foresight, and such platforms will greatly proliferate and become more probabilistic in coming years, as connectivity and automation makes them ever cheaper and easier to build. See their final report, A Journey into 2050 (2014) for insightful crowdsourced perspectives on digital futures and their policy implications.

Finally, consider Cerego. This is a learning platform that has made learning measurement, improvement, and prediction its top priority. It uses two evidence-based learning strategies, one called spaced repetition and the other called adaptive learning, to maximize the efficiency of what is learned, and to make continual and increasingly accurate predictions about what the student knows. Unfortunately, our education industry is so prediction-averse at present that companies like Cerego are few and far between, and have only recently begun to see institutional adoption.

These are only a selection of platforms that are moving humanity from random guessing to a more probabilistic view of the future. With good facilitators, group training, cognitive diversity, and good software, foresight practitioners can now build useful models of the future much faster and more robustly than the traditional research-and-report methods that have long been used by scholars, consultancies, and management teams around the world.

The Futurepedia Vision

We can see that over the next generation, social foresight will become a far more probabilistic and evidence-based activity. Because of accelerating change, significantly more of both the possibilities and the predictabilities of our future will be discoverable in advance, by some combination of web-aided human collective intelligence, statistical models, prediction platforms, and biologically-inspired computational techniques.

As part of that development, can expect the emergence of more free, open, and machine-readable collective foresight platforms. Kevin Kelly, Michelle Bowman and are among those practitioners who have independently called for the development of a Futurepedia, a free global foresight content encyclopedia and polling and prediction platform. This would include a set of crowd envisioned and improved visions of social progress, potentially of great help to social activists. A good Futurepedia would use a crowd-benefiting business model, and crowd-owned digital currency, to maximize its benefit to participants. See our Futurepedia project overview in Chapter 9 for more on that project, and let us know if you’d like to help us make it real.

The better our global sensors, computers, databases, software, communications systems, and collaborative foresight software get in coming years, the more we’ll see anticipation skills move into every walk of life. As media theorist Alvis Brigis proposes in his “total systems quantification” vision, our leading information services companies, like Google, are figuring out how to use a blend of unsupervised machine learning and human training to “tag the world”, and build increasingly accurate semantic and quantitative maps and correlative models of our past, present, and future. Once we have public semantic and quantitative maps and models that update in real-time, we’ll be able to do constant experiments aimed at improving their accuracy. Science, strategy, and collaboration will be more immediate than ever before.

Good foresight practitioners do prediction analysis and collect critical feedback on their past predictions, to measure their accuracy, to understand what methods and bias were involved, and record how much they cost to generate. They estimate how useful their best predictions would have been if they had been made well, communicated to the right decisionmakers, and acted on in a timely manner. They use this feedback to make better predictions on the trends and topics that matter most, and to tell better stories that include probabilities and predictions. We’ll offer a few of our favorite global and universal forecasts and predictions in Chapters 7 through 11.

Overdetermination: Overvaluing Probabilistic Foresight

While most of us tend to undervalue probabilistic foresight, in some contexts, as in the founding of our field in the 1940s, the opposite occurs. These future thinkers overvalue probable futures, to the neglect of both possible and preferable futures. Such thinkers will often also presume that the future is more deterministic, or fundamentally and precisely predictable, than it actually is. This cognitive bias is called overdetermination. It involves seeing only developmental processes, while ignoring the equally or more important evolutionary factors (see the 95/5 Rule). Again, see Emotional-Cognitive Biases in Chapter 2, for a brief summary of this bias, and many others.

Overdetermination is common with founders of some political, economic, social, or religious movements. We often call such individuals “Fanatics” or “Utopians”. We see overdetermination in egotistical CEOs. Often such individuals have had fantastic successes in one area, and they are guilty of assuming that everything will work the same way in a new domain. A more common kind of overdetermination happens in complacent older organizations, when they begin to assume that the future must be as it has been in the past. When strategy groups make too many simplifying assumptions about their systems, we get “oversimplification.” When we imagine future conditions being predictable further out than they are, we get “overplanning”.

Nassim Taleb in Antifragile (2012) calls overdetermination a “Fragility Complex”, and the people who engage it in “Fragilistas.” In neural network theory, we call it “overtraining”, the brittleness that emerges when neural networks (in both bio-inspired computers and in our biological minds) get too much of a specific type of training data, when the links between their nodes get too precise, and they falsely assume the future must be just like the past. In human psychology, we can call it “Old Mind Syndrome”, the rigid and increasingly brittle views of the future that can afflict us, if we aren’t careful to keep learning new things, and stay open to all the ways we are wrong. Whatever we call it, we must guard against it.

Not seeing the predictable future has led many companies into inevitable and eventually catastrophic failures. Think of all the companies, like Blockbuster or Kodak, that had years to change, but never did. But overdetermination, though it tends to afflict a smaller group of people, those with particularly large egos, advanced experience, or advanced age, has also caused some of our costliest personal, organizational, and societal failures.

Ignoring the predictable destinations ahead of us is seductive, as it seemingly frees us up to focus entirely on envisioning and creating the futures we want. But for some, seeing everything as more predictable than it is is even more seductive. When we ignore what is likely to come irrespective of our personal desires, many poor decisions, lost opportunities, wrong turns, needless waste and risk invariably occur. The same happens when we assume the future is more predictable than it actually is. When we seek too strongly to impose our vision on the world without understanding its predictable constraints, trends, patterns and cycles, we have a prescription for disaster.

Both extremes must be continually guarded against, devaluing the probable future and seeing probability where it doesn’t exist, if we are to do good foresight work. Good leaders recognize that guiding the future is a continual negotiation between what our planetary environment is delivering, whether we want it or not, what that environment will allow, and what we want. Over the longer term, what we want also has to be adaptive.