Challenge 7 – Balancing Optimism, Pessimism and Realism

How do we get better at balancing our own and our clients positive and negative sentiments (thinking and feeling) about the future? To address this key practice challenge, we must briefly visit the literature on optimism and pessimism, and understand the concept of realism, a way of thinking that uses evidence and feedback to balance optimistic and pessimistic sentiments.

Dilip Jeste, professor of psychology and neuroscience at UCSD, describes four key types of optimism and pessimism that all foresight practitioners should recognize, in ourselves and others:

1. Dispositional optimism and pessimism.
   How our personality (nature & nurture) guides our future expectations (positive or negative).

2. Strategic optimism.
   Thinking and feeling that focuses on upsides and opportunities when faced with potential risks and stresses.

3. Defensive pessimism.
   Thinking and feeling that lowers expectations and anticipates bad outcomes when faced with risks and stresses.

4. Explanatory optimism and pessimism.
   The ways we explain the causes of good and bad outcomes, and explain our ability to influence them.

Click the image for Wall Street Journal infographic. Where do you fit in to these four types?

Some of us are born and raised as dispositional optimists, others as dispositional pessimists. You may think that dispositional pessimism is somehow bad, but as Jeste explains, both orientations to the future are quite useful and adaptive in various contexts.

There is a folk saying that “pessimists are more accurate, but optimists get more done.” Sometimes what we need is more accuracy and caution, and sometimes we need more risktaking and opportunity seeking. Both approaches to our environment are needed, in different contexts.

People who are naturally more optimistic by disposition tend to be good executives, creatives, salespeople, marketers, culture leaders, and entrepreneurs, trying new things and setting stretch goals for their teams. People who are naturally more pessimistic make good deputy managers, operations, and security leaders, anticipating and heading off bad outcomes, ensuring compliance, and managing risks. These are of course broad generalizations, with many individual exceptions, but they ring true, on average, for many of us, in our personal experience.

Strategic optimism, sometimes called “ambition” or “opportunism”, is our emotional and cognitive responses that raise our future expectations and focus us on upsides and opportunities when faced with potential risks and stresses. Excitement, joy, hope, courage, and curiosity are among the dominant emotions and thoughts involved in strategic optimism. All of us have these positive feelings and thoughts, but natural pessimists exhibit them in much more limited ways.

Defensive pessimism, also called “caution” or “protectionism” , is our emotional and cognitive responses that lower our future expectations, protect us against harm, and focus us on potential downsides and problems. Fear, distrust, worry, caution, and defensiveness are among the dominant emotions involved in defensive pessimism. Again, we all have these negative feelings and thoughts, but natural optimists exhibit them in much more limited ways. Clearly, both types are useful and they balance each other, creating a more cognitively diverse world.

Whether we want to be feeling and thinking as a strategic optimist or a defensive pessimist varies by context. When you are in danger, or are responsible for security, defense, or things that may go quickly or expensively wrong, and thus need high reliability or accountability, your defensive pessimism should be in command. But if you are amid a field of opportunity, if you have a safety net, if your leader gives you protected autonomy (“top cover”) to try new things without penalty, and when the consequences of failure aren’t life threatening, you want your strategic optimism to be driving most of the time. In such situations, a positive outlook on the future will best motivate yourself, your team, and your client.

How we explain events and outcomes, our explanatory optimism and pessimism, is a product of our natural optimism or pessimism and our current feelings and thoughts, which are some a blend of opportunistic and cautious in relation to the future. Sometimes our explanations are evidence-based and adaptive, and at other times they are not. So how do we get our positive and negative feelings, thoughts, and explanations to match better with reality?

We can begin by understanding our dispositional biases, and asking whether they are adaptive or not. An often-made point is that as a species, humanity seems to still have an evolutionary pessimism, fear, or caution bias, making us naturally more pessimistic, fearful and cautious than optimistic, joyful and curious in our orientation to the world. Behavioral economist Daniel Kahneman introduces this bias, and several others in Thinking Fast and Slow (2013), a book that describes our two main neural information processing systems. What Kahneman calls System 1 is fast, older, mostly unconscious, and emotional, and System 2 is slow, newer, more conscious, and rational.

Because System 1, our intuition and emotion, is set to be both faster and stronger than System 2, our thinking, our snap judgments and emotions will quickly flood our psyches and temporarily suspend our rationality. Because we are biased so that the negative emotions are stronger than the positive ones, fear, shame, depression, and disgust will shut down our rationality much faster and longer than than joy, hope, excitement, and curiosity will strengthen our rationality. Negative emotions are simply stronger. Psychologist Daniel Goleman, in his book Emotional Intelligence, 1995 coined the term the amydgala hijack to describe a negative emotional response that is far out of proportion in its effect on our thinking and behavior to the actual degree of risk presented by the stimulus. It’s a great term. Building on this insight, we must learn to see all the ways System 1 beats out System 2 and the ways that our evolutionary biases both help and hinder us.

Dystopias Over Protopias

We must also realize that in System 1, we have much stronger set of evolutionary reasons to pay both faster and stronger attention to our negative emotions than to our positive emotions.  On average, far more of us are ready to quickly and broadly imagine dystopias (stories of regressive futures). We only slowly and reluctantly think about protopias (stories  progress-related, positive futures). It’s much easier to stand against something than for something. We’ll readily join protest marches (against something, which requires only criticism) but only occasionally join cause marches (for something, which requires an idea of progress).

Because negative emotions such as fear, danger, and caution motivate us both faster and more strongly than positive emotions like excitement, joy, and curiosity, we rarely feel the same degree of drama in stories of positive futures as we do in negative futures. Dystopian dramas outnumber and outsell protopian ones by such a large degree, it’s hard for most of us to cite good examples of the latter. Even when each offers us the same degree of absolute change, the negative potential outcome always motivates us emotionally at a deeper level than the positive one.

We likely evolved this bias, this ability to be more strongly hijacked by negative stimuli than positive ones, for good evolutionary reasons. Until very recently, we all lived in a far more dangerous and primitive world, one ruled by violence, disease, and sudden death. It was only in the late Enlightenment, the 1700’s, could the average person in the most developed parts of the world finally notice unmistakeable signs of progress over their own lifetimes. Only then did our public outlook on the possibilities for a better future start to change. So we haven’t had enough evolutionary time for our genetic proclivities to pessimism to change. As a result, biases related to this, such as loss or risk aversion bias are common in society.

There’s more to the pessimism story. A study by Lang et.al. in 2013 found that adults who have healthy doses of defensive pessimism live at least 10% longer and healthier lives. Was that an unexpected bit of information for you, like it was for me? If so, you’re probably not as in touch with your defensive pessimism as you should be. Defensive pessimists take more precautions and avoid unnecessary risks in life, rather than being foolhardy in their health habits and behaviors. This small survival benefit, along with the adaptive value of being the first to uncover potential risks, may be one of the evolutionary reasons our pessimism bias persists.

Also, the older we get, the more pessimistic we get about our own internal abilities, as they start to slow down. Unfortunately however, many aging people confuse their own naturally declining internal abilities with the generally rapidly improving external state of the world, which we can expect to get generally better faster every year, as argue in Chapters 7 to 11. They also focus on the increasing disruption that accompanies positive accelerating change, and imagine its downsides out of all proportion to the evidence and trends.

So as the average age of surviving humans continues to grow, I would bet that our cultural pessimism bias has slowly grown with it. Only 27% of the world’s population is under 25 today, and that fraction will keep going down the more developed the world gets. We might call this aging of our planet a longevity pessimism bias. It is probably less powerful than the other kinds of pessimism bias, like media bias, but it is also another factor shaping our expected future.

Furthermore, while almost all cultures are more fearful and pessimistic toward the future than they should be, some are significantly more fearful and fatalistic than others due to their recent centuries of history, as in Russia, some former Soviet republics, and parts of Africa and Asia. And of course all cultures have their “hot button” issues, that revive past traumas and reopen unhealed wounds.

These natural conditions make it easy and tempting for our media and opportunistic leaders to manipulate us for their own ends. They often play on our fears and negativity, creating media pessimism bias and opportunistic pessimism biases, each of which, if we are subject to them, push us even further from a healthy view of the world.

Any poor souls unfortunate enough to live in the agenda-driven, evidence-ignoring filter bubble of network news are constantly being assaulted with scare stories that blow all kinds of minor issues into major dramas. Those who watch only news that fits their biases can quickly become scared and dismissive of the “other”, meaning anyone who doesn’t agree with their world view.

Some of our leaders play to these fears, scaring us with extreme stories, and never pairing those stories with a sincere presentation of the latest data, which usually tell a far more nuanced and moderate view. Because of these bubbles, even as the magnitude and frequency of violence have gone down, many of us, particularly older folks, get outraged over increasingly minor differences, and don’t see the world realistically. In reality, our world is becoming, by most measures (not all) a better place.

As we’ll see in Chapter 8, filters aren’t a problem, they are a solution. We all need filters to be effective in the world, and our brain is primarily a filtering and only secondarily a thinking system. It’s main job is to figure out how to throw away too much information, continually arriving from our senses. The problem with our current filters is that they don’t learn over time, and they can’t be easily and highly customized by us, to defend our values and goals. Instead, today’s filters are top-down, outside of our control, designed primarily to further the profits of the companies offering them to us, and they don’t learn much over time. But as Personal AI comes to our filters, all of this will change.

Glassner’s The Culture of Fear, 1999, Gore’s The Assault on Reason, 2008,  and Gardner’s The Science of Fear, 2009 explore how pervasive and calculated the selling and use of fear has become in our society over the last century, and the way fear erodes our personal agency, responsibility, rationality and evidence-based approaches to reality. Messages of fear and negativity play well in a pessimism-biased culture, especially the older it gets. Our media know that “If a story bleeds, it leads.”

Because of these pessimism biases we tend focus on “headlines over trendlines”, and assume our headlines are good samples of reality (they usually aren’t). That leads us to develop poor models for how the world actually works. As Nick Kristof notes in a 2015 New York Times editorial on media pessimism bias, two-thirds of Americans today believe that the proportion of the world population living in extreme poverty has almost doubled over the last 20 years. In fact extreme poverty has done the opposite, falling by more than half over that period, falling from 35 to 14% between 1993 and 2011, per the World Bank.

All of these pessimism biases help us understand the observation by Stephen Pinker in The Better Angels of Our Nature, 2012, that even as our world has accelerated its development and become vastly richer and safer in recent centuries and decades, we adults are as or more pessimistic and worried about the future than ever before. Some of this new worry is adaptive, stemming from our increasingly fine sense of social justice, and our increasingly high value on human life as the world continues to develop. But a good deal of this worry is irrational, and must be addressed. Being highly intelligent, as some folks are, is no guarantee against pessimism bias. In fact, some of the smartest folks I know are quite pessimistic about our future, in general. They see so many ways that things are going, or could go, wrong, and choose not to see all the ways things are rapidly improving. That’s just their nature. They make great critics, but not so great entrepreneurs.

So if we carry an pessimism bias in our genes, one that often grows with age, and if the media and our leaders often exploit and amplify this bias, it is no wonder it is more fashionable to be cynical and disbelieving than to be expectant and upbeat about the future in most intellectual communities.

In sum, we can see that pessimism biases exact a steep social cost that we are just beginning to account for. They cause us to live in ways where we are more easily manipulated by fear, have poor world models, and are less open, free, healthy, and willing to risk, innovate, and experiment. Fearful and overcautious parents become overprotective helicopter parents, always “hovering” nearby to rescue their children from the first sign of minor defeat or disappointment. As Hanna Rosin reminds us in “The Overprotected Kid,” The Atlantic, Apr 2014, and Bronson and Merrryman argue in NurtureShock, 2011, children of these helicopter parents grow up disempowered, less creative, unwilling to take risks, and ready to quit as soon as things gets too difficult and don’t fit their naively overoptimistic expectations.

There’s more to the optimism story too. As Talia Sharot explains in The Optimism Bias: A Tour of the Irrationally Positive Brain, 2012, on an individual level, all of us tend to distort past and present events when we are reaching for optimist visions of the future. We forget our past and present difficulties, focus on the few positive bits that support our optimistic expectations, and imagine better and different outcomes. To some degree, these positive stories are adaptive, as Shelly Taylor explores in Positive Illusions: Creative Self-Deception and the Healthy Mind, 1991. Seeing better future outcomes can motivate us to strive harder for them. But the more we forget the negatives and limits of present realities, the more our optimistic thinking can become self-deception. Socially, too much optimism can slide into groupthink and other mass delusions, like irrational exuberance in the stock market, manias where we mimic others engaging in the same distortion, until the bubble finally bursts.

At the same time, it’s easy for any of us, for brief periods, and often in reaction to an escape from a recent bout of grinding fear and pessimism, to get carried away with our optimism, to forget the real difficulties we, our organizations, or the world have had in the past and present, and to turn our positive illusions into fantasies and manias. That usually doesn’t last long, and it isn’t very adaptive. Bipolar mood disorder and its milder cousin cyclothymia are of course both examples of cycling between too much optimism and pessimism in our moods. What we can call bipolar foresight disorder is a common affliction in a minority of futurists, in which they cycle between impractical optimism for their personal or organizational futures, then fall into serious depressions as their uncritical visions fail to realize. Both extremes are destructive, and they can’t stay on the productive ground between them for long.

Ridley (2011)

Clearly, good foresighters and futurists need to be both strategic optimists (“opportunists”) and defensive pessimists (“protectionists”) at the same time. We need to strike a dynamic balance between these two poles, and find the blend of these two feeling and thinking styles that is the most adaptive for each issue, time, and context.

We can give that dynamically adaptive blend a name: realism. Realism involves the accurate, evidence-based assessment of any system’s situation, which may be steadily improving (eg, our world), maintaining its status (many social variables), or steadily declining (eg, our physical abilities as we age). Realists don’t get too optimistic or too pessimistic about the system under consideration. They assess it accurately, by balancing both sentiments, and using evidence and feedback.

Another phrase for realism is intelligent optimism. For more, see Rohan Robert and Lara Matossian’s Intelligent Optimism community, Frank Robinson’s The Case for Rational Optimism (2009), Matt Ridley’s The Rational Optimist (2011), and Diamandis and Kotler’s Abundance (2014). Ridley’s book suffers from taking a too-uncritical look at capitalism, but each offers good introductions to the general mindset we think all evidence-based futurists should cultivate, in a world of accelerating change. A mindset that is intelligently optimistic, recognizing exponential improvements in science, technology, and innovation, but also quick to identify dangers and shortcomings of all this change, is what we need.

It is naïve and dangerous to be overly optimistic about the short term effects of new technologies, or the outcomes of many of our specific problems, in the short run. As we describe in the First Law of Tech Change, many technologies make certain aspects of society worse, at first, before better design and laws and use make them net beneficial. But it is foolhardy and ignorant not to be rationally optimistic on our prospects in general, over the longer run. In the long run we appear to be heading to a profoundly better place, as we’ll argue in Chapter 2. We get there with intelligent optimism, a step at a time.

Jeste’s Two to One (2:1) Optimism:Pessimism Thinking-Feeling Ratio: Being a Realist (Adaptive Foresight)

Let’s now ask a provocative question: Is there a roughly ideal ratio of strategic optimism to defensive pessimism in our emotions and thinking, across a broad range of contexts? In other words, an average set point we should strive for to be intelligent optimists? UCSD psychiatrist and neuroscientist Dilip Jeste has proposed that a typical healthy personal ratio, if we are not immediately in danger, may be 70% optimistic and 30% pessimistic, or roughly a 2:1 ratio. I would agree with Jeste’s guess, from my personal experience, and consider it a prescription for being a realist, a single word term we can use for adaptive foresight.

If someone asks you whether you are an optimist or a pessimist, and you say “I am a realist” this is, in my opinion, the single best word we can use for blending optimism and pessimism in an evidence-based way. Being a realist involves recognizing those things you can control, and not investing emotion in things you can’t control. At the same time, it requires an evidence-based view of life. An evidence-based view of life, as we argue in Chapters 7 and 11, presently tells us that we live in a world of accelerating complexity, productivity, and intelligence, whether we want to live there or not. Accelerating change, and the existence of apparent universal developmental trends, require us to be intelligent (balanced, regulated) optimists.

Intelligent optimism, then, is our preferred two-word description for a realist view of the world. Our choice is whether we take advantage of all the inevitable accelerating intelligence rising all around us to make a better world, using our ethics and empathy to steer us along the most desirable paths we can see. Our alternative is to get dragged into the future in an unforesighted manner, and thereby increase the chance that we, and our loved ones, will take random and potentially dehumanizing, destructive, and immoral paths to an inevitably faster, richer, safer, and stabler collective future.

Stockdale and Stockdale (1984)

Vice Admiral Jim Stockdale, imprisoned for eight years during the Vietnam War (1955-1975), and tortured fifteen times during that period, describes his ordeal in a a beautiful book In Love and War (1984), co-written with his wife Sybil. Stockdale tells us that the optimists died early, as they always expected to be released each Christmas. They became depressed and withdrawn when Christmas came and their was no change. Their expectations were repeatedly dashed, but they never updated them. Those that instead harbored small hopes, and adopted a predominantly defensively pessimistic attitude, expecting little from their environment in the near term, and focusing instead on controlling their own fear and guilt, and supporting the morale of others, were the ones that became leaders and survivors, and also thrived the best after the war. In her chapters in this book, Sybil chronicles the family pain at home during this period, and her transformation during her own ordeal from reserved housewife to proactive, outgoing leader of a POW support organization. In his collection of essays, A Vietnam Experience (1984) Stockdale further explores leadership lessons from his experiences. He observes that the greatest leaders have always held high personal ethical standards, and have also subscribed to a stoic, realist philosophy.

Stoicism, a philosophy of realism, taught by Epictetus (55-135 CE), among many others, teaches us that we should calmly and dispassionately accept and understand external events and catastrophes that are beyond our control. We should not invest emotion in them, only recognize and learn from them. At the same time, we must recognize that we are responsible for our own actions (ownership), and we can continually improve those actions by rigorous observation (awareness) and self-control. We’ll further describe this realist perspective further in Chapter 2, in our discussion of ASO (Awareness, Self-Control, and Ownership) as three necessary preconditions to adaptive personal foresight.

So if Jeste is roughly right in his guess, consider that if you are in an average environment, with no obvious dangers or fires around you that need your immediate attention, you should probably have roughly two hope and opportunity feelings and thoughts for every one fear and caution feeling and thought about the issues currently in front of you. While this ratio will surely differ from person to person and context to context, a 2:1 ratio sounds like a reasonable average set point for realistic, adaptive foresight to occur.

In a curious convergence, computer adaptive testing experts have discovered that an adaptive testing platform that serves you test questions with the conceptual difficulty adaptively calibrated so that you to get 70% of the questions correct, and 30% incorrect, is also an apparently ideal set point to maximize marginal learning per hour, as measured in post-engagement assessments, motivation to learn (average hours on the platform) and self-assessed user satisfaction with the platform. Add in a brief feedback step after any “failure”, and this may also be a rough prescription for staying in the “flow” state.

In other words, crafting your “tests”, “challenges,” and control challenges, both in the computer and in life, such that you try to succeed, in small ways, with two of your efforts for every one at which you fail, ideally again in small ways is likely to be your best training and learning regimen. Notice that this is a lot more failure than the average person wants to deal with. Perfectionists don’t like adaptive testing, even though it is the fastest way for them to improve, as it makes them fail more often than they are comfortable with.

Conversely, if you are failing more than 30% of the time, it’s probably time to scale back the reach of your personal and organizational foresight (your visions, strategies, and plans) and focus on more immediately controllable outcomes, the way a stoic returns to self-control, of emotions, thoughts, and behaviors, as the safe harbor in any storm.

We certainly need more evidence for or against all of these claims, but they are my best current model for realism (aka intelligent optimism, adaptive personal foresight), at present.

Gottman’s Five to One (5:1) Optimism:Pessimism Communication Ratio: Adaptive Relationships, in Life and in Work

Psychologist John Gottman has also published evidence-based work arguing that the ideal balance between positive and negative feelings and actions between partners in successful intimate relationships is 5:1. This is called the “magic ratio” by the press. It is also the Gottman Institute’s recommended praise to criticism ratio in the workplace.  Consider the insight these two ratios offer us. We should be more than twice as negative, in our own internal dialog, than we are in our expressions to our partners and coworkers. Defensive pessimism is important, but we must strictly regulate how we use it, or we become unhappy and inflexible naysayers, and we weaken or lose our relationships. These are deep lessons for life.

Most of Us, Unfortunately, Live in Discourse and Media Dystopias Today

So what is the typical optimism to pessimism ratio in the public discourse and media in our advanced democracies today? As I know of no formal study for the US or other countries, I’ve thought up a very rough first estimate for this variable, the ratio of majority protopian and progress-driven fiction to majority dystopian and fear-driven fiction in bookstores and movies. Quick research and personal experience tells me this ratio is roughly one to four (1:4, protopic:dystopic plots), in North America at least.

So if Jeste’s ratio is a good first estimate for an adaptive set point for intelligent optimism, and if this 1:4 optimistic:pessimistic (protopian:dystopian, hope:fear) discussion and media ratio is a good estimate of North America’s cultural ratio, this means our cultural outlook on the future is eight times worse (1:4 instead of 2:1) than is adaptive for our own minds. It is roughly twenty times worse than it should be if we wanted to treat our media as seriously as we treat our intimate friends, using Gottman’s Magic Ratio. That’s a real problem that biases our ability to see the world as it is.

We should remember that Dystopias are not just a bad futures, they are bad presents. Because we all either consume the mass media, or interact daily with others who do, we all live in Cultural Dystopias today.

An eight- to twenty-fold cultural pessimism bias with respect to our perceived futures over our actual futures, if true, is no small matter. It means we’ll be far less evidence-based and growth and progress oriented than we would otherwise be. As we’ll see in Chapter 2, our cultural pessimism bias makes it particularly hard for foresight professionals and our clients to see and take advantage of accelerating positive change. Apparently we just aren’t genetically equipped to accurately perceive or believe we live in a world of exponential improvement.

Yet all around us accelerating complexification, intelligence, interdependence, and immunity from disaster are occurring. All of humanity’s historical catastrophes have directly catalyzed the further acceleration of global complexity, intelligence, interdependence, and immunity. The future we have been evolved to imagine is not the future that is developing all around us. We are biased to not see what is actually happening, and we will eventually get beyond that blindness, one way or another. The path we take is ours to choose, but not the destination.

Of all the challenges of navigating human psychology to produce better foresight, this cultural pessimism bias may be the most important. Hilary Tindle’s book Up: How Positive Outlook Can Transform Our Health and Aging, 2013, and Michelle Gielan’s Broadcasting Happiness: The Science of Igniting and Sustaining Positive Change, 2015, are two great surveys of concrete ways to fight this bias in ourselves, our clients, and culture.

Looking ahead, we’ll need to integrate the kinds of insights Tindle and Gielan offer us into all our digital environments in coming years. Either we can do it for ourselves, or the machines will do it for us in a few decades, as their intelligence grows. Again, one way or another, I’m convinced this problem will be fixed, as it isn’t adaptive in the long run.

We won’t go further into these fascinating topics at this point. But books like Wright’s Nonzero, 2000, Robinson’s The Case for Rational Optimism, 2009, Ridley’s The Rational Optimist, 2011, Reese’s Infinite Progress, 2013, Rosling’s Factfulness, 2018, and others to be discussed later all set the record straight. Read these books if you are not yet yourself an intelligent optimist, and realize that, for reasons of biology and environment, the natural first response of most of our clients, when contemplating the future and its changes, will be some variation of fear, doubt, and pessimism first, and curiosity, courage, and optimism as weaker and slower secondary responses. This is our heritage and current condition, but it is not who we are becoming. When human nature isn’t adaptive, evolution tells us that nature must eventually change.