BROOKE GLADSTONE: Even without our keystroke patterns, we leave behind piles of identifiers ripe for collection – banking statements, phone records, Google searches. And since George W. Bush declared a war on terror, government officials and civil libertarians have battled over access to that data. In his new book, The Watchers: The Rise of America’s Surveillance State, reporter Shane Harris traces that fight, which predates 9/11, and he puts one man, John Poindexter, at the center of the government’s quest for more personal data. Best known for his involvement in the Iran-Contra Affair, Poindexter was Deputy National Security Advisor in the Reagan administration, and he argued that collecting the personal data of the masses was a crucial step in fighting terrorism. John Poindexter’s quest to do just that, says Harris, in many ways began with the 1983 bombing of Marine barracks in Beirut during the Lebanese Civil War by members of a group called Islamic Jihad.

SHANE HARRIS: It was really the first big use of suicidal religious terrorism; it really kind of introduced Americans to the idea of the suicide bomber. And, in going back and looking at sort of the reports that had been written about the attack, there were all of these clues, these pieces of intelligence in different silos of the government, some by the military, some by the CIA, a little bit by the FBI, and nobody had sort of put those pieces together. This was the beginning, not only of his personal quest to do that, but this illusive dream within government to try and connect the dots about disaster before it occurs.

BROOKE GLADSTONE: How did he manage to send us down the road while he was working in the Reagan administration?

SHANE HARRIS: So he first set out to, number one, tie the computer systems together so that the various agencies could access each other’s information. But the other thing he did was sort of a bureaucratic reshuffling. He led the creation of a number of new committees that had members from the various agencies that had a role in counterterrorism, and for the first time really got them to sit in a room together and start planning for how they would actually manage crises.

[OVERTALK] And -

BROOKE GLADSTONE: And he was really getting a lot of traction in 1984 and 1985, and then, like Icarus, his wings burn on the Iran-Contra scandal.

SHANE HARRIS: That's right. So the Iran-Contra affair, really, it’s two covert operations that sort of got sewn together, and it was his idea to put them together. The Americans were selling arms to what they believed were moderate elements in the Iranian government who were then going to try and secure the release of American hostages being held in Lebanon. So it was a sort of arms for hostages swap. Well, that operation started running a profit and Poindexter had the idea to take the money from the arms sales and to give it over to the Contras in Nicaragua, who were the anti-Communist rebel forces there.

BROOKE GLADSTONE: And Congress had cut off funding for the Contras, so this was a way to keep it going without putting it before Congress.

SHANE HARRIS: That's right. Now, Poindexter’s rationale on this was, while Congress barred the intelligence community from giving money to the Contras, they didn't say anything about the White House and the National Security Council sending them that money.

BROOKE GLADSTONE: Mm-hmm [AFFIRMATIVE]

SHANE HARRIS: So a very legalistic [LAUGHS] approach but politically disastrous, of course, because it engulfs the entire White House. He was ultimately indicted and convicted on charges, including lying to Congress, and his career is effectively ended at that time.

BROOKE GLADSTONE: So why don't we just leap ahead then to September 11th, 2001? Days after the attacks, Poindexter’s in the White House, and whatever government agency will have him, giving a PowerPoint presentation. What’s his pitch?

SHANE HARRIS: Poindexter’s idea was something he called “Total Information Awareness.” He gets this idea that, wait a second, these 19 people, the 9/11 hijackers, the 19 men who boarded the planes, were moving about in American society. Some of them were transferring money. They were renting hotel rooms. They were buying plane tickets. They were renting cars. They were doing all of these things that leave a digital footprint, but footprints that are held in private databases, commercial databases, not government databases. And what he wanted to do was get access and to marry that up with what the government already knew. Well, the government had never proposed to try and go and tap into these kinds of electronic transactions. And I remember once even asking him, I said, well, I mean, how much did you actually want to get access to in this program. And he just stopped, and he looked me, and he said, all of it. That’s why we called it [LAUGHS] “Total Information Awareness.”

[BROOKE LAUGHS] And he was just completely serious about this.

BROOKE GLADSTONE: And yet, you write about one aspect of Total Information Awareness that’s been overlooked, the safeguards to civil liberties that Poindexter actually wanted.

SHANE HARRIS: So what he said he wanted to do was invest in research of new technology that would actually take all of this data they were collecting, and the term is “anonymize” it, which means either strip out or encrypt the names and the personal identifying information. So the idea was that an analyst using this Total Information Awareness system, he sees lots of transactions, he sees patterns of activity, but he sees no names. And only if he can make a case that some pattern of electronic activity he’s seeing matches up with a pre-known pattern of terrorist activity, only then, and with the order of a judge, can he effectively unlock the encryption and see the names of the people underlying the data.

BROOKE GLADSTONE: But, nevertheless, Total Information Awareness creeped everybody out. [LAUGHS] And it was shut down.

SHANE HARRIS: Yeah.

BROOKE GLADSTONE: And basically the whole undertaking got outsourced to the NSA, the National Security Agency, and then it didn't have any safeguards at all.

SHANE HARRIS: That's right. Total Information Awareness was not a classified program. It was a research operation being run at the Defense Department. In late 2003, Congress pulls the public funding for it and then, very quietly, the National Security Agency breaks the program up into component pieces, makes them all classified. But the one piece of the program that they jettison is the privacy protection research. That was the one piece that they did not want to take over. So they throw it away.

BROOKE GLADSTONE: Meanwhile, the database that’s being amassed is rife with errors.

SHANE HARRIS: That's right. I mean, a master database that they have right now of suspected or known terrorists totals more than half a million names. Most of these are certainly false leads. The instinct here is we've been caught by surprise so many times that we just have to collect everything, and eventually we'll figure out how to make sense of it and we'll make some use of it. Well, that day has never actually come. And the analysts who are now responsible for trying to piece together the clues about terrorism are literally drowning in the data. I mean, to give you an example of just the scale of this, the National Counterterrorism Center, which is where various analysts from all of the three-letter spy agencies sit together and try and connect the dots every day, there are 28 different data networks that the analysts can get access to, and within that, 80 different what they call “intelligence streams” of unique kinds of reports. But there is no system that sits on top of all of it that allows an analyst to go to a screen, type in a keyword and immediately search all the available data. In many cases they have to go through some of these networks or databases one by one and ask them the same question. So the need for more sophisticated technology, and even just technology to lash these databases together, is really pressing.

BROOKE GLADSTONE: But Poindexter was trying to do this 25 years ago. Is institutional rivalry that entrenched that it can never be overcome?

SHANE HARRIS: I think it’s entrenched. I'm not sure it can ever be overcome, but I think that you’re hitting on it. I mean, I think that in the government, particularly in the intelligence community, which is, you know, the culture of secrecy, information is power, and the more information that you have as an agency, the more powerful you are. The less you share it, the more powerful you are. And I think it’s largely a culture that comes out of the Cold War. We're not just talking about trying to watch the Soviet Army and see how many tanks that they're amassing or how many submarines they're building. We're talking about now trying to figure out what human beings, terrorists, are going to do before they do it. It’s a completely different equation. And that mind shift that it requires to tackle that, it just hasn't happened.

BROOKE GLADSTONE: Do you think that we're in need of some sort of legal recalibration of how we gather information and how we safeguard our privacy?

SHANE HARRIS: Yeah, absolutely. The laws that we have set up to govern intelligence gathering were largely a reaction to the abuses of the '60s and the '70s where the government was monitoring people for their political activities. The laws were set up to restrict how the government collected information and under what circumstances it collected. But these were laws that were written in a time when there were comparatively few ways to actually gather data – tapping a phone, using a satellite. Today, there are countless ways that the government can collect information, either covertly through its own channels or even just by going on the Internet. So I concluded that we need to have our laws focus not so much on the collection and the acquisition of data but on setting rules and procedures for what the government actually does with those databases. And that’s a matter for Congress and the Executive Branch to take up, and they're just really not focused [LAUGHS] on that right now.

BROOKE GLADSTONE: Shane, thank you very much.

SHANE HARRIS: You’re welcome. Thank you for having me.

BROOKE GLADSTONE: Shane Harris is a reporter for The National Journal and author of The Watchers: The Rise of America’s Surveillance State.