BROOKE GLADSTONE: This is On the Media. I'm Brooke Gladstone bringing you this very special hour on data. How am I doing? Former New York Mayor Ed Koch used to ask passersby, “How am I doing?,” assembling data points from a manifestly insufficient survey. In the last segment, you heard how difficult it can be to determine which data point, what study parameter is vital and which irrelevant, or worse.

[RAP SONG UP AND UNDER]

None of this is intended to imply that we should reject data. Matt Siles of the Texas Tribune showed us how good data can lead to smart policies. I believe that the scientific process with its reproducible results and the insights they provide offers us the best chance for progress.

But science is never pure, never complete. And so it would seem wise to season the data with cultural context and values. I feel uneasy even mentioning those things, knowing full well their potential to muddy the waters.

But Jamie’s piece illustrated how data, impenetrably derived and applied by whiz kids, made dangerous situations worse in the Bronx and on Wall Street. Often we are blinded by science, so deep skepticism is warranted.

[RAP SONG/UP AND UNDER]

How much skepticism is enough? We conclude this hour with a piece that I am told will attempt to answer that question in a way that just might rock your world.

[MUSIC]

We’re joined now by Jad Abumrad, host and producer of Radiolab –

JAD ABUMRAD: Hey, Brooke.

BROOKE GLADSTONE: - who’s brought us a piece that he did involved with data. And that's pretty much all I know.

JAD ABUMRAD: Hee-hee-hee-hee-hee. So we knew you guys were doing a show on data, which apparently has  something to do with the media, but who knows what it is.

[BROOKE LAUGHS]

But data is this – first of all, it’s a great word. It’s a word that when you say, you feel rooted to the earth more strongly that you did before, ‘cause it's a word that means truth, right?

BROOKE GLADSTONE: How do we know that what we think are facts are, in fact, facts?

JAD ABUMRAD: And the question that this piece asks in a way is how do we know that the facts that were – are here today will be there tomorrow.

BROOKE GLADSTONE: Can you set it up?

JAD ABUMRAD: Sure. Well, this is a story that Robert and I bumped into.

BROOKE GLADSTONE: Robert Krulwich?

JAD ABUMRAD: That’s right, Robert Krulwich, my co-host. It just kind of spooked us a little bit. It begins with a, a guy named Jonathan Schooler, who's a psychologist, who back in the eighties did this study that everybody was just falling over themselves about. He had people watch a video of a bank robbery.   

ROBERT KRULWICH: They basically watch this bank robber walk into a bank, and he hands a note to the clerk and he says, don't press the alarm and you won’t get hurt. The clerk then hands him some money, and he exits.

JAD ABUMRAD: Now, during this whole video you see the bank robber just fine.

ROBERT KRULWICH: You get a straight on look at the bank robber, absolutely.

JAD ABUMRAD: And here is the test. After everybody watched this thing, he had half the subjects:

ROBERT KRULWICH: Write down in as much detail for five minutes everything they could remember about the appearance of the bank robber.

JAD ABUMRAD: Just to kind of help them remember. And later he had everybody, all the subjects, try and pick the guy out of a police lineup. Now, you would expect that the half of them that had to describe the guy right after seeing the video, that they would do really well at this.

BROOKE GLADSTONE: Because they’d already verbalized his physical characteristics.

[OVERTALK]

JAD ABUMRAD: Yeah, set the memory, in a sense. But, in fact, what happened was the opposite.

ROBERT KRULWICH: We found those people who had been asked to describe the face in great detail, they were actually less good –

[MUSIC UP AND UNDER]

- at recognizing the face than if they didn't engage in any description at all.

JAD ABUMRAD: And not just a little less good. They were like 30 or 40 percent less good.

ROBERT KRULWICH: So it, it was pretty whopping.

BROOKE GLADSTONE: That's amazing. And I remember this now. You guys at Radiolab did a big thing –

JAD ABUMRAD: Yeah, we did a big thing. Malcolm Gladwell did a big thing, The New York Times probably did a big thing. It was a big study. It was one of these things you were like – At dinner you were like, oh my God, you would never believe what I heard on the radio, this amazing thing.

JONATHAN SCHOOLER: Yeah, it did get – it got some press at the time, and — it really was sort of a launch pad for a, a whole set of different studies.

JAD ABUMRAD: Not to mention that it kind of made Jonathan Schooler a little bit of a rock star, you know, so to speak. And this is where story really begins because just as he was making it into all these Psych 101 textbooks, his data - started to do some funny things. And it all began when he tried to replicate that original experiment.

JONATHAN SCHOOLER: That’s right.

ROBERT KRULWICH: As you kept doing it, what happened?

JONATHAN SCHOOLER: Well, over the years, over the next five or six years, when I attempted to do it again, I would get the effect but not to the same degree that I did initially.

[OVERLAP]

JONAH LEHRER: And, and this is - this is a little troubling for him.

JAD ABUMRAD: That’s Jonah Lehrer, science writer, one of our contributing editors who’s written a lot about Schooler. And he says it went like this:  The first time Schooler tries to replicate that study, that effect?

JONAH LEHRER: Falls by 30 percent.

ROBERT KRULWICH: Ooh!

JAD ABUMRAD: The second time he tries to repeat it?

ROBERT KRULWICH: - exactly replicate what he did the first time, the effect falls another 30 percent.*

JAD: Every time he tries to redo the study, that big effect gets a little –

JONATHAN SCHOOLER: - smaller and smaller –

RICHARD KRULWICH: - this slow and downward trajectory."

[DECLINING SOUND/UP AND UNDER]

JAD ABUMRAD: It did sort of gradually get smaller. It wasn't as if all of a sudden it, it disappeared.

JONAH: Now, it's still significant. It's still publishable, but it's not nearly as exciting as it was that first time.

[MUSIC UP AND UNDER]

JAD ABUMRAD: So Jonathan Schooler’s sitting in his office and he’s thinking, what's going on? Like, how come this research started out with such thunder and then – faded? And the first theory that he really has to grapple with:

JONATHAN SCHOOLER: Is something known as a regression to the mean.

[MUSIC UP AND UNDER]

I mean, those are three of the most uninspired –

[OVERTALK]

JONAH LEHRER: Yeah.

JAD ABUMRAD:  - words put together. [LAUGHING]

JONAH LEHRER: [LAUGHING] They’re four words though.

JAD ABUMRAD: Four words.

[OVERTALK]

JONAH LEHRER: It’s regression to – the – mean.    

JAD ABUMRAD: Yeah, but Jonah, can you make it concrete for us?

JONAH LEHRER: Sure, so you flip a coin, right?  Let’s say you’ll flip a coin ten times. You may get eight heads and two tails. And you may say, oh my gosh, I've discovered a new law of coin flipping. When I flip coins in this room they’re almost always heads.

JAD ABUMRAD: But, as we all know:

JONAH LEHRER: If you kept on flipping that coin for say a thousand times, your data would show, almost certainly, unless you really had discovered something very peculiar about that room, the results would get closer to the true result, which is about 50 percent.

JAD ABUMRAD: The results would regress to the mean. And so, Jonathan's first thought was maybe that's what was happening here.

JONATHAN SCHOOLER: Yeah.

JAD ABUMRAD: Meaning:

JONATHAN SCHOOLER: When we first did the study, for whatever reason, we got lucky, or unlucky, as the case may be.

JAD ABUMRAD: You, you saw an outlier.

JONATHAN SCHOOLER: Exactly.

JONAH LEHRER: That reality is full of quirky surprises we can't explain, but over time, and this is the miracle of the scientific process, you – you regress to the true effect size.

JONATHAN SCHOOLER: But one thing about the regression to the mean account is it doesn't really explain why the effects gradually get smaller. Regression to the mean, you predict one big effect and then it should basically totter around the actual value. This gradual decline doesn't naturally fall out of the regression to the mean account.

JAD ABUMRAD: Yeah, I mean, the effect could just go away, in which case you knew you were wrong. But why would it sudden – slowly get worse?

JONATHAN SCHOOLER: Well, one possible explanation is that there was some aspect to the procedure that was important that we never really realized was important and somehow we were gradually not including whatever that secret ingredient was.

JONAH LEHRER: It could have been the color of the room in which he was conducting the experiments. It could have been how charming his grad student was, who was actually asking the students to describe the bank robber. I’m totally making up a story here.

ROBERT: Mm-hmm.

JONAH LEHRER: Let’s say that graduate student was so charming, so good-looking, so charismatic that he distracted the students. Then that grad student goes off, leaves the lab. Now he’s got a much less exciting grad student who’s not nearly as distracting. And now the effects size of verbal overshadowing has gone down.

JAD ABUMRAD: Interesting!

ROBERT KRULWICH: The only problem with that is that that little sound that Jad and I made –

[DECLINING SOUND EFFECT] –

MAN:

[MAKING SOUND EFFECT]

ROBERT KRULWICH: - means that you have to have your charming grad student at the beginning and your less charming grad student in the middle and your even less charming grad student on your

[   ? ]

JAD ABUMRAD: Yeah, they have to get slowly less charming.

JONATHAN SCHOOLER: That - that's right."

JAB ABUMRAD: And so, if you’re thinking something – is – changing here, what is it, what is it, what is it? Did you go on some kind of mad search to figure out what you might be doing differently?

JONATHAN SCHOOLER: We tried lots - a lot of different things, and in the end I just moved to another area of research. [LAUGHS]

[LAUGHTER]

ROBERT KRULWICH: You got out of town.

[MUSIC]

JAB ABUMRAD: [AND APPARENTLY, ONE OF SCHOOLER’S COLLEAGUES TOLD HIM]

JONATHAN LEHRER: --Don't worry about it, the only mistake you made was trying to replicate it in the first place.

JAD ABUMRAD: But, here's the problem:

JONATHAN SCHOOLER: It turns out it's not just me who has experienced this peculiar decline effect.

JAB ABUMRAD: As he started to look around, he realized what was happening to him - was happening all over the place.

JONATHAN SCHOOLER: In biology there was a, a meta analysis of many different biological findings showing –

[OVERLAP]

JAD ABUMRAD: There are a ton of examples, he says. And here's one he gave us:

JONATHAN SCHOOLER: In the nineties were a bunch of studies about:

JAD: – animals, using symmetry to find mates, like birds –

JAD: - females, female birds choosing their sexual partners based on -

JONATHAN SCHOOLER: - on how even their tail feathers were.

JONAH LEHRER: It was a very exciting idea. And the first year there were eight tested, at – and all eight found, yep fluctuating asymmetry - that's what the phenomena is called – is real. We also got an effect size.

JAD: So it seemed true.

JONAH LEHRER: Yeah, and they’re all over the world, in all these different species; females had evolved this unconscious tendency to prefer symmetrical males. The next year it's tested 12 times, and 9 of the 12 confirm it. And then things start to fall apart.

[DECLINING SOUND EFFECT]

You can make the sound effect – yeah, exactly.

[LAUGHTER]

Until by the end of the ‘90s you're going 1 for 13.  

JAD: One for thirteen?

JONAH LEHRER: One, one for thirteen. Now –

[OVERTALK]

ROBERT KRULWICH: Now, of course these studies are not black white, yes no studies, there’s some gradation, but this was the basic trend that Jonah saw.

JAD ABUMRAD: And just in case birds seem a little distant, here’s another example:

JONAH LEHRER: And I think this is, for me, the most  troubling error to the decline effect, ‘cause you see like second generation anti-psychotic.

ROBERT KRULWICH: Second generation anti-psychotic.

JONAH LEHRER: These are drugs used to treat people with schizophrenia, bipolar. When they first came out in the late ‘80s, early ‘90s, some studies found that they were about twice as effective than first generation anti-psychotics.

ROBERT: Mm!

JONAH LEHRER: And then what happened is the standard story of the decline effect. Cue the sound effect.

[SOUND EFFECT]

Which is clinical trial after clinical trial, the effect size just slowly started to fall apart. You see a similar decline with things like Prozac, and – anti-depressants –

JAD: Wow!

JONAH LEHRER: The effect of the drugs have gotten weaker, but the placebo effect has also gotten stronger. I was talking to one guy at a drug company who [LAUGHS] – he was kind of interesting. He blamed that on drug advertising. He said that they started to see their placebo effect go up in the late ‘90s when these drug companies started advertising.

JAD ABUMRAD: But then wouldn’t that actually offer an explanation for this decline thing because, you know, some – if you know about what this drug is supposed to do, maybe it works differently somehow?

JONATHAN SCHOOLER: Certainly, there are areas of psychology where that can change the outcome in, in one way or another, but it's very unlikely that, you know, in say these female preferences for symmetrical feathers that, you know, the birds got wind –

[MAN LAUGHING]

- of this symmetry finding and now all of a sudden they’re not – not into it anymore.

[LAUGHTER]

JAD ABUMRAD: I don’t know.

ROBERT KRULWICH: If you haven’t been around a chickadee conversation lately –

JAD ABUMRAD: [LAUGHS] Yeah, it passes quickly amongst the chickadees.

ROBERT KRULWICH: Can you explain why what you found at the beginning, isn't that what you find now?

JAD: And why is - why it gradually r — you know, went away?

[OVERTALK]

ROBERT: Yeah, why it gradually went away.

JAD: The “gradually” is still puzzling.

JONATHAN SCHOOLER: I tell you, it - I'm personally - baffled.

JONAH LEHRER: It's tough to come up with an all-purpose explanation or some easy fix.

JAB ABUMRAD: It could be a lot of things, says Jonah. I mean, in some cases it might be statistics.

JONATHAN SCHOOLER: Regression to the mean is almost sure to be a part of it.

JAD ABUMRAD: Or maybe it's, you know –

JONATHAN SCHOOLER: It could be this gradual change in the procedure in something that we just don't know what it was that happened.

JAD: You can't rule it out.

JONATHAN SCHOOLER: But – I would probably be less shocked than most people if something unconventional is actually involved in this, as well.

ROBERT: Unconventional, like?

JONATHAN SCHOOLER: Like – I, I say this with - some trepidation but I think we can't rule out the possibility that there could be some way in which the active observation is actually changing the nature of reality.

[MUSIC/MUSIC UP AND UNDER]

That somehow in the process of observing effects, that we change the nature of those effects.

[MUSIC]

SEVERAL AT ONCE: Ah!

ROBERT: You’re in real trouble.

[GUYS LAUGHING]

JAD: But it sounds like you're saying that maybe the truth is running away from you or something?

JONATHAN SCHOOLER: Well, I - I'm not – you know, I’m not gonna say that I am certainly not gonna say that there's some sort of intentionality to these effects disappearing, more that it's almost – and again, this is just a speculation - some sort of habituation.

So just as when you put your hand on your leg you feel it and then as you leave it there it becomes less and less noticeable, somehow there may be some kind of habituation that happens in - with respect to these findings.

JAD: But what is the hand and what is the leg in – in having this –

JONATHAN SCHOOLER: Well, in - in - in this most radical conjecture, there could be some sort of collective consciousness that's habituating. Again, radical speculation but there may be some peculiar way about the nature of reality that somehow it gets into the ether.

[MUSIC]

Keep in mind, the notion that the laws of reality are unchangeable is an assumption. It’s a reasonable assumption but we don't know it for a fact. And there have been physicists who have even speculated that perhaps the rules change as time – goes on.

[MUSIC]

JAD ABUMRAD: But by this logic you can never actually know anything for sure –

ROBERT KRULWICH: Because reality could change based upon the observer’s position, habits, biases, information whatever.

JONATHAN SCHOOLER: Well, so far we have not really seen these types of things in the domain of, of physics, but you know an aspirin might not do what it used to.

MAN: Huh!

JONATHAN SCHOOLER: There's a question that you haven't - asked, which is let's say that we were to do a study  and demonstrate this decline effect, that when you keep running experiments, that they get smaller? Well, what happens when you try to replicate that effect?

[LAUGHTER]

Does the declined effect decline?

[GUYS LAUGHING]

JAD: Yeah, that’s a good question.

[LAUGHTER

JONATHAN SCHOOLER: Maybe we could just get rid of the decline effect by studying it.

[LAUGHTER]

[MUSIC/MUSIC UP AND UNDER]

BROOKE GLADSTONE: Where does that leave certainty?

JAD ABUMRAD: Well, first of all, it doesn’t happen everywhere, this effect that we’re talking about. Gravity, you know, Newton's Law and a all that stuff seems to be constant. But it does seem to happen in places where there are a lot of variables at play. Science, of course, as you know, is this thing where you try and isolate everything except one thing. And you change that one thing, and you study the effect that that one change has.

So this may be an indication that if you’re not isolating that one thing, that there are many causes that you’re not aware of and that we're not measuring what we think we're measuring? I kind of like his theory though.

[LAUGHTER]

JAD ABUMRAD: Me, being the person who likes to believe in magic –

[MAN LAUGHING]

I — I, I sort of go with – go with his formula.

BROOKE GLADSTONE: I just don't understand how the human ether is going to affect the sexual preferences of birds.

JAD ABUMRAD: The birds perhaps have their own collective unconscious, which –

[BROOKE LAUGHS]

- comingles with ours in the ether, where the radio waves go, you know, and the spirits reside. So it could be up there somewhere.

[MAN LAUGHS]

BROOKE GLADSTONE: Do you think you contributed, Jad, personally to this disturbance in the ether by making such a big deal of this finding?

JAD ABUMRAD: I think we did yeah, yeah. I mean, the media is biased, and I mean not in the way that people think it is, but it's certainly biased towards tension, it’s biased towards surprise. And so, there might be some kind of bias that leads us all towards a result that is counterintuitive and exciting.

BROOKE GLADSTONE: Yeah, but I mean it's not as if you didn't do due diligence –

JAD ABUMRAD: No, we did.

BROOKE GLADSTONE: - or that Dr. Schooler didn't do due diligence.

[OVERLAP/BOTH AT ONCE]

JAD ABUMRAD: That’s what’s weird.

BROOKE GLADSTONE: How many times do you have to perform a study?

JAD ABUMRAD: That's a really good question. I mean, the lesson that I draw from all this is that in certain places, like in that whole school of psychology studies about people doing things that seem really counterintuitive and that are really fun to talk about -  you know what I'm talking about - you can smell them in the air, that – that kind of study.

You just have to think to yourself, wait a second, is it really that simple. There may be something spooky in the universe or it may be that the scientific process is, in some sense, undermined by all the things we don't even know we don't know about. But it's just really hard. You just have to be careful.

BROOKE GLADSTONE: The bottom drops out of your show if you can't quote a study because it hasn't been reproduced 500 times.

JAD ABUMRAD: Well, that’s – just means that we’re going to be a sports show.

[BROOKE LAUGHING]

Which has been my secret desire all along.

BROOKE GLADSTONE: And our program leaves people with the message we love to leave people with:  Trust…

[LAUGHTER] 

No one. Thanks again, Jad.

JAD ABUMRAD: Of course, thank you.

[MUSIC/MUSIC UP AND UNDER]

BROOKE GLADSTONE: That's it for this week's show. On the Media was produced by Jamie York, Mike Vuolo, Nazanin Rafsanjani, Alex Goldman, PJ Vogt and Sarah Abdurrahman, and edited by me. Our technical director is Jennifer Munson, with help this week from Dylan Keefe.

Katya Rogers is our senior producer. Ellen Horne is WNYC’s senior director of National Programs. Bassist/composer Ben Allison wrote our theme. You can listen to the program and find transcripts at Onthemedia.org. You can also post comments there. You can find us on Facebook or follow us on Twitter, and you can email us at Onthemedia@wnyc.org.

On the Media is produced by WNYC and distributed by NPR. Bob Garfield will most definitely be back next week. I'm Brooke Gladstone.

*Correction: An earlier version of this short incorrectly stated that Jonathan Schooler saw the effect size of his study fall by 30% on two different occasions. In fact, he saw it fall by that amount the first time he repeated the study and saw a general downward trend thereafter. The audio has been adjusted to reflect this fact.

**Correction: An earlier version of this short incorrectly attributed a statement to Jonathan Schooler’s advisor. The statement was actually made by his colleague. The audio has been adjusted to reflect this fact.