Brooke Gladstone: From WNYC in New York, this is On the Media. I'm Brooke Gladstone. With 2023 swiftly coming to a close, many have turned their political sites on the coming new year with the November calendar boasting a presidential election. Grab those tarot cards, Yarrow stocks, look it up, or magic eight balls because America's quadrennial guessing game is underway.

Female Speaker 1: A turn now to a new polling of a hypothetical 2024 election matchup that shows Donald Trump beating President Joe Biden in seven swing states. Trump holds a four-point lead over Biden in Arizona, a six-point lead in Georgia, four points in Michigan, three points in Nevada, two points in Pennsylvania, four points in Wisconsin.

Male Speaker 1: According to the Decision Desk HQ on the general election fraud, Trump is topping Biden by 2.3 percentage points overall.

Brooke Gladstone: Polls and polls and polls, the prime ingredient in horse race coverage even as we media critics clamor for the nourishment of policy reporting, but it cannot be denied. Biden does face a challenge known to many and incumbent a public pretty ticked about the present and worried about the future of the next four years at least.

Male Speaker 2: A new CBS News poll is showing most Americans disapprove of President Biden's handling of the Israel-Hamas war, with few thinking his administration's actions are bringing things closer to a peaceful resolution.

Male Speaker 3: It's not just the war. 7 in 10 Americans say they disapprove of the president's handling of inflation and immigration at the border.

Female Speaker 2: CNBC out with a new all-America economic survey, this one looking at the Biden administration losing some support heading into election year.

Male Speaker 4: President Biden, terrible approval numbers no matter how you look at it.

Brooke Gladstone: Of course, Biden's campaign woes are very different from his projected opponents. In fact, it's not clear his projected opponent would concede any woes because look at the numbers.

Female Speaker 3: Donald Trump is defending his recent declaration to be dictator as he still holds a commanding lead in the polls.

Male Speaker 5: A new Siena College/New York Times polling that shows Republican voters favoring him by 54 to 17% over his nearest rival, Ron DeSantis.

Male Speaker 6: There were 26 different demographic groups that got broken down. You know how many of those groups Donald Trump didn't lead amongst?

Male Speaker 5: No.

Male Speaker 6: Zero.

Ruth Igielnik: What we've seen in our data so far is that there's an increased share of people who say that Trump has committed serious federal crimes and in fact, an increased part of that group says that they're likely to continue to vote for Trump.

Brooke Gladstone: Ruth Igielnik is the staff editor for News Surveys at the New York Times. She was part of the team that crafted a New York Times/Siena College poll that came out this week.

Ruth Igielnik: Other polling organizations have compared the four different possible trials and there's no question that the two trials around election interference, the January 6th trial, and the Georgia trial are seen as more impactful, more meaningful for voters overall and for Trump voters. We definitely saw in our data and in conversations with voters that, for example, the case in Florida around the documents at Mar-a-Lago is far less concerning for a lot of Trump voters in particular.

Brooke Gladstone: The polls show Trump beating Biden by two points and Biden's approval on Israel is especially low, especially with the young people.

Ruth Igielnik: There's no question that in our data across age groups, we saw low approval for Biden on how he's handling Israel, from young voters all the way to the top. We did ask who voters trust more on the Israeli-Palestinian conflict and there we saw voters saying that they trust Trump by a pretty decent margin to handle the situation in Israel over Biden, and particularly young voters by nearly 20 points were more likely to say they trusted Trump than Biden. Older voters by about seven points said that they trusted Biden over Trump, but the public overall said that they trust Donald Trump more to handle a conflict in Israel.

Brooke Gladstone: Let's talk about the way a question is framed because it can mean anything and it can mean everything. Were there debates about the wording of any questions in these particular polls?

Ruth Igielnik: In any poll, but in particular, this poll, we wanted to be very careful and conscientious in how we worded the questions because we know that the conflict between Israel and Gaza right now is very complicated, and even that choice of words I had now is really complicated. We had to really think carefully about when we're saying Gaza, when we're saying Palestinians, when we're saying Israel, when we're saying the Israeli government.

Each of those decisions was very, very careful and with any poll, we spend a lot of time thinking about question wording with two main goals. The first is that a question feels fair to anybody who's being asked it. We want to represent every single voter in America, no matter their opinion, no matter where they sit on the political spectrum. That means that every voter in America should be able to look at these questions and say, I see myself in those answer choices, there's an answer choice there that represents how I feel. That requires a lot of debate.

The second thing that we spend a lot of time doing is trying to ensure that the questions are interpreted the same by everybody who's answering them. Oftentimes with polls, we can get something what's called measurement error where we think we're measuring something but different respondents interpret it different ways. We again, spend a lot of time being very careful and very precise.

Brooke Gladstone: Can you give me an example?

Ruth Igielnik: Yes. We wanted to understand this idea of a ceasefire. If we just asked, are you in favor of a ceasefire? We might get a lot of measurement error there. Some people think a ceasefire is all the fighting stopping now. Some people might think a ceasefire is the fighting stopping as soon as let's say the hostages are released, or any number of conditions.

Brooke Gladstone: Or stopping forever with no condition.

Ruth Igielnik: Or stopping forever, right, with no conditions. Just asking ceasefire is too broad a word that lots of people have different interpretations. For me, as an analyst of polls, I would know how to interpret that data because everybody's bringing their own perspective to it. Instead, we had a forced choice option, either Israel should stop its campaign in order to protect against civilian casualties, even if not all the hostages are released, or Israel should continue its military campaign until all the hostages are released, even if it means civilian casualties in Gaza might continue.

Now, a lot of people might disagree with those answer choices and I think it's very reasonable to disagree with those as the only two answer choices but in doing that, we actually saw that the public is fairly split on that question, which I think is interesting. I think a good survey question does split the public more evenly.

Brooke Gladstone: Did you consider having more than two choices?

Ruth Igielnik: We did. We debated that pretty heavily and there were times where we got close to having that as options. In the end, it makes it a lot more difficult to analyze the data, the more options that you provide. Our goal here was this as a first volley into these things and we find something interesting, we can ask additional questions and try to tease apart those groups as we go.

Brooke Gladstone: How many people did you poll overall?

Ruth Igielnik: We polled a little over 1,000.

Brooke Gladstone: A typical number, but on the smaller side, right?

Ruth Igielnik: This is actually a current fallacy I think of polling in an issue for debate. Some polling is traditionally around 1,000 adults or voters nationwide. There's a statistical principle called the central limit theorem and that says, when you get to a certain sample size, you have less variation over time, and 1,000 is around that sample size for US adults. While there are a lot of online surveys these days that have much larger sample sizes, I always stress to reporters, a larger sample size isn't necessarily better. In fact, you can get a pretty good result out of 1,000 voters and it's not that much better to have a lot more.

Brooke Gladstone: Do you regard the recent poll you helped to craft as an election poll or an issue poll?

Ruth Igielnik: We had questions that were with the goal of trying to understand where the electorate stands right now on the upcoming election but we also took an opportunity in this poll to really dive deeply on two issues. On the situation in the Middle East and then on Trump's trial so this one kind of straddles both worlds.

Brooke Gladstone: Which one presents more problems in trying to get an accurate result?

Ruth Igielnik: I think they're both very challenging to get an accurate result, let me say that to start. Election polling is really difficult because we are trying to predict a population that doesn't exist yet, we're trying to predict who is likely to vote, and that's a group of people that don't know if they're going to vote, they can change their minds, there are a lot of things that happen between now and then and so that is a very challenging feat. Issue polling is also really challenging for a different set of reasons. Issue polling the challenge is more of that question development challenge.

Brooke Gladstone: Since we're going into an election year, Ruth, I think we could all use a refresher on polling literacy so let's run through a few things that listeners might want to look out for. First of all, how accurate are polls? I'm guessing it's harder to get people on the phone these days and many who do answer the phone aren't necessarily interested in responding to a pollster.

Ruth Igielnik: Yes. I think it's reasonable to be concerned about poll accuracy. One of the things that we in the polling community track is the average error of polls over time so how far election polls are off from the final results, and that's actually getting smaller, over time the polling results are getting closer to the actual election result with certainly some outliers, some elections polling fares worse. For example, we've historically seen that polling is less accurate when Donald Trump is on the ballot than when he's not. For example, in 2022, The New York Times/Siena College poll, we in all of the races we polled we were within a couple of points of the final result. We were very accurate.

Brooke Gladstone: Then again, a couple of points in a country so divided can make all the difference.

Ruth Igielnik: Yes, you're absolutely right. One thing I really struggle with is that polling is really a blunt instrument and it's not always a great instrument for measuring elections. To be able to get within two points feels pretty good but it's obviously not good enough to truly show us the outcome. You asked specifically about getting people on the phone and it's a really good question because it is something that we as an industry struggle with. I remember when I started in the polling world response rates meaning the rate of people who respond and answer a poll were around 10%. That number is now around 1% or 2% of people actually responding so it's very low.

Brooke Gladstone: Yet you say polls are more accurate now than they used to be.

Ruth Igielnik: One of the things that we and others in the industry do are what we call non-response studies. We try to understand if people who don't respond to polls are fundamentally different than people who do respond to polls. That's really where the difference happens. If the people who respond to polls are different in their attitudes and opinions, then there's cause for concern. What we're seeing so far is the people who do respond to polls even if it's a small sliver of the population, they're not different in their attitudes and opinions than people who don't respond to polls. We're also exploring other methods beyond just phone polling. People have moved more towards online polling or some kind of hybrid between phone and online polling.

Brooke Gladstone: Those aren't randomized, right?

Ruth Igielnik: Correct, exactly. Then there is a third category that a lot of pollsters have explored and this is this idea of an online poll that is recruited over the phone. They contact people over the phone and then they bring them into an online panel so it is random, and these are what some people call probability-based panels. There's a lot of smart people in the polling industry who are trying to answer these questions because it's a concern for all of us. Making sure that we are still accurate and representative is the most important thing to us.

Brooke Gladstone: We had sociologist Zeynep Tufekci on the show back in 2020 and she compared election forecasts to weather forecasts. When it comes to election forecasts. She said--

Zeynep Tufekci: There's a lot of evidence and reporting that shows that the forecast itself affects the outcome. That's where they differ from weather forecasts. If I don't take an umbrella, the rain is not going to happen just to spite me, whereas if I think something's more likely to happen and act in a particular way, I may actually be changing the odds it's going to happen.

Ruth Igielnik: There's a particularly interesting overlap between people who are interested in election forecasting and interested in weather forecasting in the polling community and I find it hilarious. I think it's a good question to ask. I would draw a distinction between polls and averages of polls and then what some people layer on top of it which are these probabilistic models that suggest how likely someone is to win.

I'll give an example. Biden is down by two and then they say, and that means that Biden has a 20% chance of winning. It's that chance of winning that I think is really challenging for people to interpret. You can imagine in 2016 when models said, for example, Hillary had a 70% chance of winning a lot of people mentally rounded that up to 100% and then felt that she was guaranteed to win. I think a lot of the behavioral changes that came out of this weren't just because of polling but were because of these probabilistic models that were layered on top of polling.

Brooke Gladstone: Behavioral changes meaning?

Ruth Igielnik: If people decided to not vote or there was some reporting that people were choosing to switch their votes, any of those kinds of decisions they were often because of this idea that they thought somebody had a 70% chance of winning and so their behavior didn't make a difference in that outcome.

Brooke Gladstone: Whereas the 30% chance of losing is still a big chance?

Ruth Igielnik: Huge.

Brooke Gladstone: [chuckles] If there were a 30% chance of rain you would ponder over bringing an umbrella?

Ruth Igielnik: Yes.

Brooke Gladstone: What are the top three things you'd like news consumers to keep in mind when interpreting poll responses?

Ruth Igielnik: One, don't focus on any single poll. Any single poll could be an outlier so looking at multiple polls over time is hugely valuable. The second thing is be smart about what polls you're consuming. The metric I like to use to actually assess whether we think a poll is good quality or not really doesn't require any methodological understanding, it's just whether or not a poll is being transparent about their methods.

If you look at some of these higher quality pollsters they will go deep and explain their methods and everything that they're doing. That's what we're looking for. A lot of these lower quality pollsters they're putting out a short couple-sentence statement that explains what they found without a lot of information about how they did it. That should be a red flag that that might not be a poll that you should give a lot of credence to.

Number three is to not ignore poll results that you don't like. That actually means you should probably pay more attention to it. For example, with our Battleground Polls last month, we had a lot of people push back on the idea that Biden was losing and he was particularly losing among young voters and Black and Hispanic voters. That was a poll result that a lot of people didn't like, it wasn't necessarily representative of the people around them. At the same time. That's a real change that's happening in this country and many polls are showing that not just us and so I think it's good to pay attention to those results, not ignore them, especially if they are happening across multiple polls.

Brooke Gladstone: Ruth, thank you very much.

Ruth Igielnik: Thank you so much for having me, this has been great.

Brooke Gladstone: Ruth Igielnik is the staff editor for new surveys of The New York Times. Coming up, that feeling when you really want to avoid the news. Given that, hey, thanks for listening. This is On the Media.