BOB GARFIELD: From WNYC in New York, this is On the Media. I’m Bob Garfield.

BROOKE GLADSTONE: And I’m Brooke Gladstone. So now we are in the stifling dog days of summer, tied by the ancients to the rising star of Sirius, the Dog. The ancients called the dog days “evil days.” Homer said they brought fevers to suffering humanity. John Brady wrote in the 1800S that these days soured wine, maddened dogs and afflicted man with diseases, hysterics and frenzies. Do you feel it? I ask because you, like many Americans, may be struggling for some cooling comfort, refreshing predictions of better times to come. Don't count on it. Prognosticators, especially political ones, are so often so very wrong. Remember?

[CLIPS]:

MALE CORRESPONDENT: Jeb Bush is the strongest general election nominee, by far. He’s got the best message for a general.

MALE CORRESPONDENT: He’s gonna raise the most money. The establishment likes him the best. A lot of members of Congress would like to see him be the nominee. And, obviously, to some extent, it’s his turn in the primogenitor of the party.

BROOKE GLADSTONE: Ooopsie!

FEMALE CORRESPONDENT: Well, it is game over for Jeb Bush, after disappointing results in South Carolina.

BROOKE GLADSTONE: Then there was Bernie Sanders.

MALE CORRESPONDENT: An adorable socialist but probably not, you know, a serious candidate.

BROOKE GLADSTONE: But, um –

MALE CORRESPONDENT: Senator Sanders did an unbelievable job, let’s, let’s be honest. He brought the party in the right direction. He excited people. He had good policy positions that are actually built into the platform that will be on the floor tomorrow.

BROOKE GLADSTONE: And the earlier reactions to the future GOP nominee.

FEMALE CORRESPONDENT: Trump is America’s summer fling.

MALE CORRESPONDENT: This is a phenomenon that is going to fade sooner rather than later.

FEMALE CORRESPONDENT: I think last night is the beginning of the end for Trump. That is my prediction.

BROOKE GLADSTONE: Ah, halcyon days.

DONALD TRUMP: I humbly and gratefully accept your nomination for the presidency of the United States.

[CROWD APPLAUSE & CHEERS]
BROOKE GLADSTONE: Because of this terrible, horrible, no good, very bad year for the talking heads, we've updated and are re-airing a show that was first broadcast in March, all about the perilous art of political prognostication because, even in the best of years, media predictions rarely beat a magic 8 ball.

BOB GARFIELD: Phil Tetlock is a professor of psychology and management at the University of Pennsylvania and is an expert on prognostication. He’s conducted two landmark studies that uncovered the finer points of forecasting, the first, Expert Political Judgment: How Good Is It, How Can We Know? and more recently, Superforecasting: The Art and Science of Prediction. Phil, welcome to On the Media.

PHIL TETLOCK: Thank you.

BOB GARFIELD: People like you break down prognosticators into two general categories.

PHIL TETLOCK: You know, there’s an old joke that there are two types of people, those who divide the world into two types of people and those who don't.

[BOB LAUGHS]

But you’re right. I talked about foxes and hedgehogs in the early work and I think it's a critical distinction. It helps to be a little more fox-like, a little more self-critical, a little more eclectic and willing to draw ideas from lots of different places. You pay a price for committing to one point of view and trying to assimilate everything into it. And that's more or less what the hedgehogs do.

BOB GARFIELD: As you flip around your cable channels, do you see more foxes or do you see more hedgehogs?

PHIL TETLOCK: The hedgehogs outnumber the foxes by a very substantial margin, and there's a pretty obvious reason for that. The hedgehogs give the media the sound bites that they want. Hedgehogs have one big idea. They try to assimilate everything into that big idea, and that makes for a compelling narrative package. Foxes are more likely to interrupt their flow of thought with things like “but, however, although.” So that led me to propose one very simple rule of thumb for distinguishing more fox-like from more hedgehog-like pundits. Foxes are more likely to say “however” and hedgehogs are more likely to say “moreover,” so you have the however over moreover ratio.

BOB GARFIELD: [LAUGHS] Hedgehogs, you say, are historically less reliable. That’s not just a qualitative judgment that you've made. You know their batting averages.

PHIL TETLOCK: We do. We keep the track records over extended periods of time. One of the reasons why the more hedgehog-like forecasters have bad batting averages is that they’re so aggressive, intellectually aggressive. They’re like batters who are swinging for home runs all the time and they strike out a lot.

BOB GARFIELD: Oh, it’s like Ryan Howard of my Phillies. He hits about 229 with power, but there's an ongoing question of whether he's any longer useful to the team. Are these guys, these hedgehogs, useful to the team or, more to the point, are they useful to democracy?

PHIL TETLOCK: Well, there’s a big illusion that develops over time. The hedgehogs are swinging really hard. They’re making confident judgments. They’re often assigning pretty high probabilities to low-frequency events. If you do that often enough, sometimes you're going to be right and you’re going to be hauled out in front of the media as “here's the guy who predicted the rise of Islamic fundamentalism, here's the guy who predicted the Russian invasion of the Ukraine or the Syrian civil war or the whatever, because they’re predicting so much of the time. Like the broken clock theory - the broken clock is right twice a day - there will virtually always be some extremist hedgehogs who are going to be well positioned to claim credit for predictive successes.

BOB GARFIELD: Now Phil, this hour was occasioned in part because it seems that nearly every elite political pundit, and some not so elite ones, like - me, got Trump completely wrong. And it seems that if we believe we knew anything about our nominal areas of expertise, we in the punditocracy have learned our lesson. Jack Shafer, the media critic at Politico, called Trump a black swan. Tell me, what is a black swan?

PHIL TETLOCK: The idea comes from the fact that Europeans didn't know about the existence of black swans until they got to Australia. Black swans were sort of unthinkable, and that's a metaphor for things that are radically unpredictable; nobody saw them coming but, after the fact, everybody has an explanation for them.

BOB GARFIELD: What makes this guy such a black swan that he could have so flummoxed just everybody in the prognostication arts?

PHIL TETLOCK: Donald Trump is not a totally black swan. He’s more of a dark gray swan. There were people in the summer of 2015 who thought he had some low probability chance. It may have been 5%, 10%. And over time, they’ve gradually increased that probability as Trump has proven to be much more politically durable than virtually everybody in the know thought.

Here's the bum rap that a lot of pundits get. Let’s just say, for the sake of argument, that you're a wonderful pundit and you're well calibrated, which means that when you say that something is 90% likely, that thing happens 90% of the time, you say 70%, that thing happens 70% of the time. And something you say is 90 or 95% unlikely, happens, does that mean you're automatically a terrible forecaster? No. Sometimes unlikely things happen. If you're a perfectly calibrated forecaster and you say something is 95% likely, 5% of the time you should be wrong.

BOB GARFIELD: Now Phil, as you know, our partner in this project is Nate Silver and FiveThirtyEight.com. It does gather all of the available data and contemporaneous polls and historical data and, say, under these conditions, 73% of the time so and so will win. But Nate didn’t see Trump coming, either. Do you have any thoughts on this?

PHIL TETLOCK: Sure. Nate Silver and his team at FiveThirtyEight are extraordinarily adept at analyzing poll data. There really weren't a lot of quantitative data to crunch about Donald Trump in the summer of 2015. The question in this situation isn't, you know, did you see an extremely gray swan from a long distance, which very few people are ever going to be able to do. The question is, how quickly do you change your mind as new evidence comes in? So between the summer of 2015, when Donald Trump was, indeed, a very dark gray swan, and now, when he is not at all, what were the cues, what were the clues that led Nate and other people to change their minds?

Note, by the way, that if the question in the summer of 2015 was, will Trump be the nominee or will Trump be the next president, it’s not at all clear that either of those things will happen, but the likelihoods of each of those things has materially increased. The question is by how much and how we go about making those determinations, and it would be interesting to hear someone like Nate talk about that.

BOB GARFIELD: I think there’s a 95% probability [LAUGHS] that you’re going to get to do that very thing. Phil, thank you so much.

PHIL TETLOCK: My pleasure.

BOB GARFIELD: Philip Tetlock is a professor of management and psychology at the University of Pennsylvania. He is co-leader of the Good Judgment Project, which you can find at gjopen.com.

BROOKE GLADSTONE: Actually, make that 100%. Here's Nate Silver's answer.

NATE SILVER: The short answer is that there were a lot of moments, right? And you can kind of see us going from saying 2% to 5% to 10% to 20% to 50%. Now I’d say it’s at higher than 50%. That gradual adjustment is, at least by the textbook, consistent with what you're supposed to do, right? You're supposed to start out with what we might call a prior belief. You say, candidates like Donald Trump don't usually win party nominations and here are the reasons why. Then you adjust that view against the weight of the evidence and when you see things that are part of your theory that, hey, the party will organize to stop Donald Trump and this is not happening in October, it's not happening in December - it is happening now but may be too late.

[MUSIC UP & UNDER]

So we did update our view over the course of the campaign.

BROOKE GLADSTONE: Coming up, more on how polls and money influence election predictions and why the best answer this year may be “Reply hazy, try again.”

BOB GARFIELD: This is On the Media.