BOB GARFIELD: Even when there are no brutal dictatorships to bring down, the digital revolution brings skirmishes every hour of every day. Some are obvious – the Mac versus the PC, Kindle versus Nook, the movie industry versus pirates, hackers versus websites, viruses versus your own hard drive. But one of the biggest battlefronts of the Internet is largely out of view. It is the endless clandestine struggle for the strategic high ground on Google. So dominant is the search giant that a placement of a search result on the first page can be the difference between success or failure, in retail sales, and a lot more. For example, one study has shown that one-third of users click only on the top result. Another third clicks on results two through five. If you’re listed somewhere beyond the first page, you will be very lonely. These are differences measured in the billions of dollars. For the rest of this program, we will examine the stakes of and the ongoing guerilla war over what Google called PageRank, and we will examine the art and science of manipulating those rankings, a practice called SEO, search engine optimization. Our journey begins with the most valuable piece of software ever created. It’s called the Google algorithm, and it’s hard to define because, like the National Security Agency budget and the formula for Coke, it is top secret. We know that it ranks search results based on contextual relevance. If you type the words “dog” and “food,” you'll get a lot of results for dog food. But to choose which dog food-related website to list first, Google also counts how many other web pages have linked to it. Those links reflect what the Internet at large thinks is the most relevant site, and their measurement is Google’s special sauce, that and another 200 factors, which the company doesn't publicize and which it tinkers with every single day. The goal of SEO is to exploit what is known about the Google algorithm to improve search rank. This can be as simple and benign as tagging a blog post with keywords embedded in the text or as sinister as rigging the results, which is precisely what JCPenney did in the weeks leading up to Christmas. According to New York Times business reporter David Segal, Penney's was landing at the top of Google results in product category after product category.
DAVID SEGAL: For about three months they were turning up number one or sometimes number two for dozens of very valuable search terms, things like “dresses” and “furniture,” really generic commercial terms, and sometimes more obscure things, like “grommet top curtains,” which I wasn't aware of before. But, yeah, there was this massive campaign, and it was very effective.
BOB GARFIELD: Organically, JCPenney would not have come up at the very top of a Google search for “grommet-topped curtains” and “furniture.” Obviously, something was afoot. What was afoot?
DAVID SEGAL: There are two important ways that Google ranks. There’s actually several hundred factors in their algorithm, but two of the important ways - one is just the relevance of the word itself, the sort of anchor text, as they call it - and then the second very important factor are the number of links to your website. Essentially it’s like voting. So if you have a website dedicated to, say, folding paper airplanes, the more people that are fans of folding paper airplanes that link to your website, the higher your website will rank when people enter the term “folding paper airplanes.” What JCPenney did, or perhaps its search engine consultants did, was they set up an elaborate campaign to make it look as though JCPenney had been linked to by thousands of different websites all over the world for key search terms that they cared about. So, one of those terms was “dresses.” And, in fact, there were 2015 websites around the world that had linked to JCPenney. You would go to these websites and you would see, out of the blue, the word “dresses” or “little black dress” or “casual dress,” and you would hit and you'd bounce right to the JCPenney website. And these were completely irrelevant to clothing or to dresses. There was one that was on nuclear.engineeringaddict.com.
[BOB LAUGHS] One [LAUGHS] was on [BOB LAUGHS] casino-focus.com. There were websites in Russia about dentistry. But this campaign of links and having those links there really worked.
BOB GARFIELD: Tell me how Google reacted once your story broke.
DAVID SEGAL: They really took it all in, in stride and kind of emphasized that this is what they do, and this was sort of another day at the office. And, at the same time, the punishment that they’d doled out to JCPenney was swift and brutal. I remember I left, I think it was a Wednesday night to go to dinner, and at six o'clock I just checked for the couple dozen search terms that I'd been looking at for JCPenney, and they were all number one or number two. And when I got back at about 7:30 they had been buried. For dresses, for instance, which I'd followed pretty carefully, they were number, I think, 73, and for a bunch of other terms they were in the 40s and the 50s. And really, when you get off that first page, you’re gone. And I was feeling just the kind of awe at the power of Google. Just so much of commerce goes through this one website. And a bunch of guys in a room, led by this one guy, Matt Cutts, gets to decide who has broken the rules. And if you break the rules, you’re in serious trouble.
BOB GARFIELD: I'm curious if the algorithm at this point is now kind of like the U.S. Tax Code or Windows, so layered on with various kinds of fixes and patches and add-ons that it has long since mutated far away from its original purity.
DAVID SEGAL: The answer to that is that nobody really knows. Google says they change their algorithm about 500 times every year. There are lots of people that are constantly testing and figuring out what those changes mean. So Google would say that they're making the, the product better. But the sort of sobering reality of looking at JCPenney is that there still are ways to fool this system.
BOB GARFIELD: David, thanks so much.
DAVID SEGAL: It was my pleasure. Thank you.
BOB GARFIELD: David Segal is a business reporter and, increasingly, the SEO cop for The New York Times.