The Limits of Expertise

Last month Tom Nichols, professor of national security affairs at the U.S. Naval War College  and a well regarded authority on Russian foreign policy and American nuclear strategy, published a thought-provoking essay on his blog titled “The Death of Expertise:”

1I wonder if we are witnessing the “death of expertise:” a Google-fueled, Wikipedia-based, blog-sodden collapse of any division between students and teachers, knowers and wonderers, or even between those of any achievement in an area and those with none at all.

By this, I do not mean the death of actual expertise, the knowledge of specific things that sets some people apart from others in various areas. There will always be doctors, lawyers, engineers, and other specialists in various fields.

Rather, what I fear has died is any acknowledgement of expertise as anything that should alter our thoughts or change the way we live. A fair number of Americans now seem to reject the notion that one person is more likely to be right about something, due to education, experience, or other attributes of achievement, than any other.

Indeed, to a certain segment of the American public, the idea that one person knows more than another person is an appalling thought, and perhaps even a not-too-subtle attempt to put down one’s fellow citizen. It’s certainly thought to be rude: to judge from social media and op-eds, the claim of expertise — and especially any claim that expertise should guide the outcome of a disagreement — is now considered by many people to be worse than a direct personal insult.

This is a very bad thing. Yes, it’s true that experts can make mistakes, as disasters from thalidomide to the Challenger explosion tragically remind us. But mostly, experts have a pretty good batting average compared to laymen: doctors, whatever their errors, seem to do better with most illnesses than faith healers or your Aunt Ginny and her special chicken gut poultice. To reject the notion of expertise, and to replace it with a sanctimonious insistence that every person has a right to his or her own opinion, is just plain silly

I encourage visitors to the Stage to read Dr. Nichol’s entire piece. It was prompted by what has become a common experience every time he (or fellow UNWC professor and former NSA employee John Schindler) decides to publish a new essay or speak publicly about a pressing issue of the day. Soon after his work is published a flood of acrimonious tweets and e-mails follow, declaring that he does not really understand how American intelligence agencies, the Kremlin, or the Obama administration actually work. 

Most of these responses are misinformed. Many are simply rude and mean. They are not an impressive example of what laymen commentators can add to America’s political discourse. Dr. Nichols suggests four rules of thumb for engaged citizens that he believes would improve matters:

1.The expert isn’t always right.

2. But an expert is far more likely to be right than you are.

3. Your political opinions have value in terms of what you want to see happen, how you view justice and right. Your political analysis as a layman has far less value, and probably isn’t — indeed, almost certainly isn’t — as good as you think it is.

4. On a question of factual interpretation or evaluation, the expert’s view is likely to be better-informed than yours. At that point, you’re best served by listening, not carping and arguing.

2

The trouble with this advice is that there are plenty of perfectly rational reasons to distrust those with political expertise. Mr. Nichol’s wiser readers, for example, may have heard of the research conducted by Philip Tetlock, presented most compellingly in his 2005 book, Expert Political Judgement: How Good is it? How Can We Know?. Louis Menard describes the results of Dr. Tetlock’s research program in a book review for the  New Yorker: 

“Tetlock is a psychologist—he teaches at Berkeley—and his conclusions are based on a long-term study that he began twenty years ago. He picked two hundred and eighty-four people who made their living “commenting or offering advice on political and economic trends,” and he started asking them to assess the probability that various things would or would not come to pass, both in the areas of the world in which they specialized and in areas about which they were not expert. Would there be a nonviolent end to apartheid in South Africa? Would Gorbachev be ousted in a coup? Would the United States go to war in the Persian Gulf? Would Canada disintegrate? (Many experts believed that it would, on the ground that Quebec would succeed in seceding.) And so on. By the end of the study, in 2003, the experts had made 82,361 forecasts. Tetlock also asked questions designed to determine how they reached their judgments, how they reacted when their predictions proved to be wrong, how they evaluated new information that did not support their views, and how they assessed the probability that rival theories and predictions were accurate.“3

Tetlock’s experts came from all sorts of backgrounds: included were media personalities, tenured academics, professional analysts in Washington think tanks, and employees of numerous government agencies (including those with access to classified materials). A wide range of political beliefs and styles of analysis were also included: the study included both registered Republicans and Democrats, Austrian economists and their Keynesian counterparts, specialists in game theory, realist IR, area studies, and every other analytic model that gained popularity during the test period. What were the results of their 82,000 predictions?

“The results were unimpressive. On the first scale, the experts performed worse than they would have if they had simply assigned an equal probability to all three outcomes—if they had given each possible future a thirty-three-per-cent chance of occurring. Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys, who would have distributed their picks evenly over the three choices.

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight…. The expert also suffers from knowing too much: the more facts an expert has, the more information is available to be enlisted in support of his or her pet theories, and the more chains of causation he or she can find beguiling. This helps explain why specialists fail to outguess non-specialists. The odds tend to be with the obvious” (emphasis added). 4

In other words, the expert is not more likely to be right than you are! Rigorous examination of actual expert track records show the opposite to be true: the more famous, confident, and specialized an expert is, the less accurate their political judgments tend to be. It turns out what an expert truly excels at is explaining away his or her own horrible record. Dr. Tetlock describes what happened at the end of the study when participants were asked to explain their errors:

“When  we recontacted experts to gauge their reactions to confirmation or disconfirmation of their predictions, we frequently ran into a awkward problem. Our records of the probability judgements made at the beginning of the forcast periods often disagreed with experts recollections of what they predicted. Experts claimed that they assigned higher probabilities to outcomes that materialized than they did. From a narrowly Bayesian perspective, this 20/20 hindsight  effect was a methodological nuisance: it is hard to ask someone why they got it wrong when the think it is right. But from a psychological perspective the hindsight effect is intriguing in its own right… , so we decided in six cases, to ask experts to recollect their positions prior to receiving the reminder from our records. When we asked experts to recall their original likelihood judgments, experts, especially hedgehogs, often claimed that they attached higher probabilities to what subsequently happened than they did…. Experts [also] shortchanged competition. When experts recalled the probabilities they once thought their most influential rivals would assign t the future that materialized, they imputed lower probabilities after the fact than before he fact. In effect, experts displayed both the classic hindsight effect (claiming more credit for predicting the future than they deserved) and the mirror image effect (giving less credit to their opponents for anticipating the future they deserved.)“ 5

When confronted with these truths very few experts would admit that their reasoning or their methods were wrong. Instead they would create elaborate justifications for each failed prediction, claiming that their predictions were “almost right,” that events had truly been “close calls,” or that their prediction was only thrown off by some “out of the blue” or “fluke” occurrence no one could have seen coming. Each failed prediction was successfully turned into a compelling story that not only justified the expert’s failures but made things seem as if they and their methods had been right all along.

Mr. Tetlock’s research deserves to be better known than it is. In addition to Louis Menard’s review in the New Yorker excerpted above, interested readers are encouraged to read the CATO Unbound issue devoted to the book, view Tetlock’s hour long presentation for the Long Now Foundation, or purchase the book itself. It is difficult to delve into his work and think about experts the same way again.

Of course, Mr. Tetlock is not the only person to put our faith in expertise to question. The last decade has seen a general assault on the role of experts in a wide variety of fields: research by John Ioannis has shown that much of–if not most of–the medical findings published every year are false, and Daniele Fanelli has used similar methods to critique scientific research as a whole.6Doctors like Ben Goldcare and Adrian Fugh-Berman have shown how expert medical practitioners are often influenced by both common human biases and industry connections to prescribe treatments not in their patients best interests.7Popular public intellectual Nassim Nicholas Taleb made his claim to fame tearing down the models and methods of Wall Street’s top financial experts, maintaining that all narratives used by experts to explain the world are post-hoc constructions used to hide their own lack of knowledge.8  James C Scott has shown how, despite any amount of study, training, or advance knowledge managers are given, the complexity of the systems expert bureaucrats are asked to manage are often beyond understanding, and attempts to merely comprehend (much less manage) these stewardship are doomed to distort and weaken them. 9

This list is not comprehensive: it is limited to a few of the more stunning examples that have emerged over the last decade (and does not include all of those). But in many ways this is a very old critique. Men like Friedrich Hayek, Hermann Kahn, and Paul Meehl made similar points (or in Meehl’s case, conducted similar studies) decades ago.10 Their skepticism has deep roots in Western culture; the Western tradition of free inquiry began with a man condemned to death for exposing how little his society’s elites actually knew. Thoughtful people have been rejecting expert opinion for two millenia. 

All of this leaves us with an important question: if experts are no better at making sense of the world than the rest of us, what are they good for?

While this post has expressed some hostility towards expert judgment, I am not an intellectual Jacobin intent upon getting rid of our experts altogether. The best evidence suggests that there is a valuable role experts can play. Indeed, some expertise does not need to be questioned at all. In their review of recent research on expert intuition, psychologists Daniel Kahneman and Gary Klein summarize the findings of James Shanteau‘s “cross-domain” study of expert performance:

“The importance of predictable environments and opportunities to learn them was apparent in an early review of professions in which expertise develops. Shanteau (1992) reviewed evidence showing that [real, measurable] expertise was found in livestock judges, astronomers, test pilots, soil judges, chess masters, physicists, mathematicians, accountants, grain inspectors, photo interpreters, and insurance analysts. In contrast, Shanteau noted poor performance by experienced professionals in another large set of occupations: stockbrokers, clinical psychologists, psychiatrists, college admissions officers, court judges, personnel selectors, and intelligence analysts.” [11

Mr. Kahneman and Klein conclude that an expert’s judgement is most valuable in predictable environments whose patterns can be recognized, memorized, and then internalized. They call these “high validity” domains:

“Validity, as we use the term, describes the causal and statistical structure of the relevant environment. For example, it is very likely that there are early indications that a building is about to collapse in a fire or that an infant will soon show obvious symptoms of infection…. In contrast, outcomes are effectively unpredictable in zero-validity environments. To a good approximation, predictions of the future value of individual stocks and long-term forecasts of political events are made in a zero-validity environment….Validity and uncertainty are not incompatible. Some environments are both highly valid and substantially uncertain. Poker and warfare are examples. The best moves in such situations reliably increase the potential for success.”  12

Alas, “zero-validity environments” are exactly the kind environments at the heart of this discussion with. Zero validity environments tend to fall into two categories:

 1) Environments whose size or complexity make it impossible for experts to recognize the patterns or relationships they need to understand in order to make valid judgements about the system (such as faced by economists, ecologists, and financial analysts).

2) Environments where experts must evaluate behaviors, attitudes, past history, and other personal idiosyncrasies to try and explain why individuals act as they do or how they will act in the future (such as faced by psychiatrists, college admissions officers, and court judges).

“Political analysis,” which usually attempts to explain how statesmen, politicians, soldiers, or other political actors will respond to or instigate changes in world affairs, the electoral arena, battlefields, or some other political environment, falls into both categories! The expert political analyst is expected to create a reliable psychological portrait of key decision makers and understand the the numerous contingencies and complexities of the environment in which these decisions makers work. Unfortunately for the expert, this task lies beyond the limits of human cognition.  No human, no matter how learned, experienced, or credentialed, can overcome them.

This seems pretty dismal, but as I suggested earlier, our experts need not lose hope. Their advanced degrees were not attained in vain. Too many have simply been proclaiming their expertise in the wrong subjects. Experts have no advantage over the average reader of the New York Times when it comes to political judgements. They are no better–and sometimes worse–at predicting political outcomes, weighing the relative importance of events, or identifying cause and effect relationships than the rest of us. But analyst experts do have a significant advantage over the layman analyst in one domain: they know so much more. Experts can contribute a great deal to the conversation simply by telling us what they know.

An easy way to distinguish the difference between these two types of expertise is to think in terms of what and why.  Experts are at their best when they explain the “what” of the systems they have spent so much time studying. This kind of expertise is about knowledge, not judgment or analysis; its explanations are not prescriptive or predictive, but descriptive. They focus on confirming disputed facts and explaining the significance of details that layman observers do not have the knowledge to properly understand or recognize. Their job is to find the truth and make it known.

This task is far more important than it initially seems. Consider the ‘red-line’ kerfuffle that made headlines back in September. Several weeks before serin filled munitions were launched into a suburb outside of Damascus called Ghouta. The President of the United States had previously declared that if the Assad Regime used chemical weapons then the United States would intervene in the conflict. The world was left with a question: were these weapons launched by the Syrian Army or one of the groups opposing it?

In those days opinions on the subject flew off the presses. Most of this was political analysis par excellence–lots of talk about the various groups’ “motives” or “credibility,” and the “political logic” of the situation. But as the literature reviewed above suggests, once a certain level of quality was reached, none of these analyses were more reliable than the others. We had ran straight into the limits of human political judgment.

Fortunately, there was a more promising route to answering the question at hand. At the time I was closely following Eliot Higgins’s Brown Moses Blog. Mr. Higgins, now recognized as an expert in the munitions used in the Syrian war,13  put together all of the videos and pictures he could get that were in some way tied to the scene of the attacks and compared them to other munitions used in the conflict. His careful reconstruction both identified the exact munitions used in the attack and linked them to artillery and munitions used by Assad’s regime.

Mr. Higgins here accomplished something only an expert could. The number of Americans who have the background knowledge necessary to identify and distinguish the dozens of munitions used by Syrian militias is limited to a few dozen experts. These men have a true “expert’s advantage” that comes from gaining specialized knowledge in the what of an issue.

This advantage is not limited to the world of material objects. A recent post at Andrew Chubb‘s website is an excellent example of how expertise can be properly applied to human subjects. The question at hand: to what extant do the views articulated by vocal PLA ‘hawks’ like Luo Yuan, Dai Xu, and Zhang Zhaozhang  represent the upper levels of the People’s Liberation Army as a whole? (I encourage those interesting in following this debate in full to read Mr. Chubb’s original essays for the Jamestown Foundation’s China Brief, a more recent review essay by Yawei Liu and Justine Zheng Ren for for the Journal of Contemporary China, and Chubb’s response to this essay at South Sea Conversations). This is essentially a debate about a fairly narrow “what”: what are the past and current institutional roles of the hawks? Are they in a position to know and articulate PLA strategic doctrine? 14

As with the munitions in Syria, analysts who approach this question need a thorough command of things not commonly known: the structure of the PLA, the biographies of the officers in question, PLA-media relations procedure, and the ability to recognize those damn Chinese characters used in all of those press releases and interviews. Only those with a high level of expertise can even participate in this debate. Those of us who lack this knowledge  are well advised to take Dr. Nichols’s advice to stop our “carping” and listen to what these folks have to say.

One suspects that the unpolished masses who flood Dr. Nichol’s and Schindler’s inboxes with insults and assertions do not do so because they have been reading Philip Tetlock’s most recent research. While we have reviewed plenty of compelling evidence that experts and expert claims should be viewed with skepticism, the great majority of this research is unknown among even the more educated sections of the body politic.  What accounts for this widespread hostility towards expertise?

Dr. Nichols suggests the following:

Some of it is purely due to the globalization of communication. There are no longer any gatekeepers: the journals and op-ed pages that were once strictly edited have been drowned under the weight of self-publishable blogs (like, say, this one). There was once a time when participation in public debate required submission of a letter or an article, and that submission had to be written intelligently, pass editorial review, and stand with the author’s name and credentials attached. Even then, it was a big deal to get a letter in a major newspaper.

Now, anyone can crowd the comments section of any major publication with inane blather. We live in a huge high school boys’ room, where anyone with a marker can write anything on the wall. Sometimes, that kind of free-for-all spurs good thinking. Most of the time, it just means people can post anything they want, under any anonymous cover, and never have to defend their views or get called out for being wrong.

Another reason for the collapse of expertise lies not with the global commons but with the increasingly partisan nature of U.S. political campaigns. There was once a time when presidents would win elections and then scour universities and think-tanks for a brain trust; that’s how Henry Kissinger, Samuel Huntington, Zbigniew Brzezinski and others ended up in government service while moving between places like Harvard and Columbia.

…I also would argue that colleges have to own some of this mess. The idea of telling students that professors run the show and know better than they do strikes many students as something like uppity lip from the help, and so many profs don’t do it. Many colleges are boutiques, in which the professors are expected to be something like intellectual valets. This produces nothing but a delusion of intellectual adequacy in children who should be instructed, not catered to.

…Finally, the root of this collapse of standards lies in our manic reinterpretation of “democracy,” in which everyone must have their say, and it must not be, er, disrespected. (The verb to disrespect is one of the most obnoxious and insidious innovations in our language in years, because it really means “to fail to pay me the impossibly high requirement of respect I demand.”)15

It is unfair to blame all of America’s ills (or even the specific ones mentioned above) on our hapless experts. 16But the truthfulness of the narrative is irrelevant. What matters is how people who live far away from the ivory towers and government offices see the experts who dwell there. And in in this kind of political climate the experts who craft political analysis defending the powers that be should not be surprised when they are met with  unrestrained hostility and extreme skepticism. It is the cost of claiming expertise.   

Doug Sosnik does a fair job describing the new environment  that all people who claim some special authority their fellow citizens lack now find themselves:  

“Americans’ long-brewing discontent shows clear signs of reaching a boiling point. And when it happens, the country will judge its politicians through a new filter—one that asks, “Which side of the barricade are you on? Is it the side of the out-of-touch political class that clings to the status quo by protecting those at the top and their own political agendas, or is it the side that is fighting for the kind of change that will make the government work for the people—all the people?17

It is not just the politicians that will be forced through the grinder. Everybody who talks politics or claims a position of special authority is going to be, or is already, facing this scrutiny. People are wondering what side of the barricade America’s credentialed class of experts belong.