In a recent post, I talked about how Google is the master manipulator of the press. I was planning to follow that up with a 10 year case study of how brilliantly Google has managed the media.
However, as so often happens on the Web, I was distracted. This time, it was an e-mail from a friend. He is what I call a “regular” person, not an “Internet marketer.” We tend to forget the gigantic chasm between “them” and “us.”
However, there he was, commenting on content farms and their incredibly ridiculous material.
When regular people start noticing Google flaws, red alarms blare at Google. New “algorithms” are released to much fanfare. The “searching masses” are reassured and Google’s all-important public perception is, once again, safe.
Collateral damage of “the fix”? It is merely the cost of doing business. Google does not even pay that price. The “little guy” does.
Back to my “regular” friend and his e-mail about content farms. The insane examples of what he sent reminded me of a search I had recently done for “how to convert cable (i.e., coaxial) to HDMI.”
It turned up inane ehow.com articles. They are masterful lessons to all high school students in how to write “filler.” But they were mostly low-value (i.e., useless), regurgitated content (what we call “pap”). I recalled many other eHow (and wikiHow) articles I had found when searching at Google.
It amazed me that Google could rank eHow’s pap so highly. Was it just my bad luck to keep finding them and their filler articles?
No, I had read many articles by the cognoscenti about the “no-value” quality of ehow.com’s content, all wondering how Google ranked their pages so highly. Some examples of those posts…
In a July 2010 article in pbs.org, one of ehow.com’s contributors sums it up…“I was completely aware that I was writing crap… Never trust anything you read on eHow.com.”
Forbes reported a matter that is more serious than pap in January, 2011, plagiarism at eHow.
Blekko, a new search engine, had even outright banned eHow articles from its search results.
As word of the impending Google anti-pap algorithm circulated, just about every knowledgeable observer expected that ehow.com (the crown jewel of a company that went public in January, 2011 at a valuation of over $1.4 billion) would be one of the hardest hit. Heck, they were the “poster boy of pap.”
So what happened?
Google’s algorithm did decimate many well-known content farms.
Somehow, it missed ehow.com. Their traffic actually increased by a stunning 20%.
How Did eHow Escape “The Farmer”?
How could eHow’s traffic not only survive, but increase?
Google refused to give an explanation. Barring that, one can only conclude that eHow was given a pass for some reason, that they were “whitelisted.”
Why no explanation from Google? Google says that it prefers not to talk about individual cases.
Actually, Google seems to pick its spots. It was happy to announce and talk about its penalty of BMW when it discovered BMW’s black hat tricks.
Rather than follow my “Master Manipulator” post with the 10 year case study on Google’s consummate media management skills, I wrote the post on Google Farmer’s false-negatives (bad content that was not penalized), asking readers to submit examples.
Read the sample search results that were provided by readers in the comments. They are sadly hilarious.
If ever there were articles that Google Farmer should have hit hard, it is the majority of those in ehow.com’s content farm. Many could serve as the very definition of pap. Google’s new algorithm was “made” to “seek and destroy” these. And yet…
Ehow.com’s traffic actually increased. Given that some content farms dropped by 50-90+%, the conclusion is compelling…
The Power of Whitelisting
Whitelisting has many meanings. In the context of search, it is the practice of making exceptions by manually adding sites to a list that would exempt them from specific signals or changes in the algorithm.
It was, and is, the only answer that seems to be a reasonable explanation of this and other Google actions.
However, Google has always denied whitelisting. They denied using it to correct Farmer errors, too.
But how else do you reasonably explain how the “Cult of Mac” was suddenly and miraculously re-instated?
The owner was lucky. As a former managing editor at WIRED, he was able to reach WIRED.com with his story. He complained (March 1) about how Farmer caused “traffic to the site to fall by one-third to one-half over the weekend.”
And then, BINGO!… “The site is miraculously back. Everything looks great.” (Scroll down to the UPDATE at the end of the link to the WIRED article above.)
A miracle, indeed. It sure looked like a manual change.
That, of course, caused Google to attempt to head that conclusion off at the pass. Google rushed (March 2) to release a statement through Search Engine Land that there were no significant changes, manual or otherwise, to “Farmer,” only “the usual smaller tweaks.”
However, minor tweaks could not result in such a dramatic reversal of fortune. No other sites seem to have benefited from those “tweaks.” It seems to be a miracle for one site.
“Miracles” don’t happen in the computer world of bits and bytes. In the world of Google, they do tend to happen when publicity is loud and negative. And sure enough, within 24 hours of its owner going public, Cult of Mac was “miraculously” back to normal.
Results and circumstances speak louder than words. In the same WIRED article, read Google’s words…**** “We deeply care about the people who are generating high-quality content sites, which are the key to a healthy web ecosystem. However, we don’t manually change anything along these lines.”
The only reasonable interpretation actually appears to be…
“We care more about the algorithm and our margins than the people whose lives we dislocate, unless of course one of those people knows whom to reach in order to generate some significant publicity. Then, miracles happen.”
Then, just a week later, something happened that almost is a miracle…
Google Admits It!
On March 10, 2011, Danny Sullivan reported that Google admitted the existence of a whitelist during a panel discussion at an SEO conference. Some excerpts from that post…
Google has “exception lists for sites that might get hit by some algorithm signals.”
“Sites might be excluded from the impact of particular ranking signals.”
Google sent an official statement to Sullivan, which was later included as a postscript. It admitted that they use “exception lists when specific algorithms inadvertently impact websites, and when we believe an exception list will significantly improve search quality.”
The Register covered that story, noting that…
“Google has admitted that it uses whitelists to manually override its search algorithms, more than a year after its European corporate counsel denied the existence of whitelists when defending the company against antitrust complaints in the EU.”
It notes that Matt Cutts, head of Google’s anti-spam team, added that…
Google will make an exception for sites it believes have been wrongly demoted on its search pages.
The Register went on to explain how this was in direct contradiction of a statement by Google to EU antitrust regulators that it “doesn’t whitelist or blacklist anyone.” Google had repeated statements to the same effect to The Register as late as 3 months ago (December, 2010).
The Register also provided evidence of another website, Foundem, that had been damaged by another algorithm change (it had been removed from Google’s search results). It suffered for 3 years. It only returned to Google’s organic listings after “it went public with its story at the end of 2009.”
And The Register explores how Google tries to stage-manage issues, right down to terminology. In the same way “Farmer” (name given by others) became “Panda” (name later given by Google – warmer, cuddly, no matter what Google’s back story for the name is), “whitelisting” became “exceptions.” It’s worth reading the entire article.
The bigger picture is that, over and over again, we will see the use of manual changes when publicity is hot. Google has done its best to deny it, except in extreme circumstances. But there is no point in hiding it any longer, not when it’s obvious. Suddenly, Google not only admits “whitelisting”…
Matters Turn Black
Less than one month earlier (February 14, 2011), Google admitted that it uses blacklists, manual changes that penalize companies that successfully manipulate Google’s rankings to reach the top of search results. Matt Cutts reveals in a video…
In other words, Google does not only evaluate individual sites and cure the hurt (whitelists), it also manually puts the hurt (blacklists) on others.
Why admit blacklisting after years of denial? They had no choice…
The New York Times had, just two days earlier (February 12), broke the JC Penney story of successful black-hatting of many Google search results for search terms such as “bedding” and “area rugs.”
The most stunning aspect of that story? It’s hard to see how Google’s sophisticated algorithm could miss a massive, systematic and commercial manipulation. This was far from subtle, and not even close to being technically sophisticated.
The explanation? In an interview with the NY Times (same article), Matt Cutts said…
“Given the one billion queries that Google handles each day, I think we do an amazing job.”
Google does, of course, do an amazing job of organizing the Web’s information. (This series of posts is not about Google’s quality, it’s about the ethics of their communication strategy.)
This was a miss of epic proportions. Cutts’ statement is red-herring damage control.
This time, Google could not finesse their way out with vague explanations. They admitted to taking “manual action,” demoting JC Penney in the search results. The New York Times reports…
“At 7 p.m. Eastern time on Wednesday, J.C. Penney was still the No. 1 result for “Samsonite carry on luggage.”
Two hours later, it was at No. 71.
At 7 p.m. on Wednesday, Penney was No. 1 in searches for “living room furniture.”
By 9 p.m., it had sunk to No. 68.
In other words, one moment Penney was the most visible online destination for living room furniture in the country.
Then next it was essentially buried.”
And then, the inexplicable happens..
Google Wriggles Off Again
Cutts’ answer (“so many searches, so few people”) begged the following question, which they did ask…
WHY did Google fail to catch a campaign that had been under way for months? One, no less, that benefited a company that Google had already taken action against three times? And one that relied on a collection of websites that were not exactly hiding their spamminess?
Cutts emphasized that there are 200 million domain names and a mere 24,000 employees at Google. “Spammers never stop.”
And that is when the NY Times lets Google off the hook.
The important fact is not that there are 200 million domains (Cutts failed to mention that only half are active sites). Nor is it important that they have a “mere” 24,000 employees to handle 1 billion searches per day (the employees are not actually doing the searches and sending the answers back!).
Here’s the key fact…
Google generated $2.5 billion in profits (on total revenues of $8.4 billion) in the last quarter of 2010, up 29% from Q4 2009 during a recession. The vast majority is still generated from advertising, either through Google-owned sites or AdSense.
“The algorithm,” and Google’s fierce defense of its algorithmic automation and stubborn refusal to make hand changes (except to quieten negative publicity), enables it to earn those astronomical margins (30%).
Rather than challenge Google on that basis, the story withers away out of the public’s view.
We know that the mass media have no fear. It pushes and probes until major companies and politicians are revealed for what they are, not for what they’d like us to believe. And yet, somehow, they are bamboozled by answers like, “there aren’t very many of us here.”
They need to understand that “Don’t be evil” was dropped as its corporate motto two years ago. In that article, they talk of not having an “evil meter,” and yet this Techcrunch article explains how CEO Eric Schmidt used an “evil scale” to justify entering the Chinese market.
It seems that Google did not like being held up to such a high standard. So they replaced it with “You can make money without doing evil,” giving examples that redefine “evil” extremely narrowly.
Note to the mass media… govern yourselves accordingly. You are not talking to Google circa 2003.
In this context, the correct challenge to Google was the following…
“Why don’t you use a tiny part of those profits to recognize and correct the damage you do to hard-working, honest online businesses?” In other words…
Whitelist and Blacklist When You Are Not Under Public Pressure
A careful analysis of Google’s history of communications (my next post) reveals that Google corrects injustices only when under public pressure. Otherwise, it hides behind “the algorithm.”
Google’s answer, when it is not under the New York Times’s magnifying glass, is that it’s not them…
“It’s the algorithm, stupid.” (We call it “ITAS” for short.)
To the best of my knowledge, there exists no private (i.e., unpublicized to any significant degree) case where Google manually alleviates the gut-wrenching financial pain of good sites unfairly penalized due to Google error, sites invited by Google to submit their stories here.
Quite the contrary. None of the sites who are reporting their pain at the link above are featured stories in WIRED or the NY Times. But read their stories. Their pain is the same as the Cult of Mac’s.
Until the press makes Google’s social responsibilities “a story,” Google will continue to be “unable to make manual exceptions” (as outlined in Google’s invitation) for the 900 (and counting) sites that have submitted their stories to date. Google will continue to ignore the responsibility of fairness that a near-monopoly (on search) inherits.
Nor will Google privately blacklist people or companies who earn income, often large numbers of AdSense dollars, by manipulating their way to the top of the search engine results by breaking some or all of Google’s guidelines (except for one site known to us – that Webmaster made the fatal “mistake” of publicly teaching others how to spam Google, all the while snubbing his nose at Google).
The Only Hope for Fairness?
Media with substantial public reach and stature, reporters from the “A-list press,” must ask why Google does not institute the obvious next step, to grow the blacklist substantially with a massive and well-publicized program. The same, of course, is true for the whitelist.
Google supporters say that it’s not realistic, that such a model does not “scale.” But “scale” is merely Google-talk for making more money by enforcing “ITAS” as much as possible.
In fact, Google could, with the program outlined at the link above…
generate and maintain a blacklist with tens of thousands of domains on it
reduce black-hat attempts dramatically
develop more useful data to improve the algorithm.
(To those who would rebut… No, this is not the same as Google’s “in the field” QA program.)
A similar program to develop a comprehensive whitelist would cover errors on both sides of the algorithm.
Naturally, each list would shorten (over time) for two reasons…
fewer attempts due to the nature of the program
ever-decreasing gap between “algorithm” and “actual” as the algo improves, enabling the removal of domains.
How much would this social fairness and the improvement in search results cost Google?
A decimal point on that $2.5 billion in profits.
After years of denials, we now know that both whitelists and blacklists (“manual changes”) exist. History shows that they are used primarily to neutralize negative publicity (i.e., the public loss of confidence in Google’s search quality).
Despite this new knowledge, Google will continue to deny the use of manual changes to fix similar issues in the future (except under the most obvious of circumstances, in which case it can now admit the use of a tool that it has acknowledged).
There is, unfortunately, no reason to believe future denials. Google has spent their credibility. Self-preservation drives changes, not fair play.
There is nothing fair when Google reverses the fortunes of such sites as Foundem (whitelist), Cult of Mac (whitelist but denied), JC Penney (blacklist), and eHow (a total mystery) while ignoring the plight of so many other companies.
These other companies are smaller, buried in the cloak of “this is not news” privacy. But they are either…
every bit as devastated (good sites penalized by Google error) and should be whitelisted until the algo improves
getting away with financial murder, a theme we are all too used to seeing on Wall Street (when does someone say “enough”?).
Google says (in the Sullivan whitelisting post) that they “go to great lengths to apply our quality standards and guidelines fairly to all websites.” How can that be, when over 900 sites have already reported themselves as being negatively impacted? There are, of course, thousands more.
Those sites will be hurting for quite a while. On that page, the preamble by a Google employee states…
“Note that as this is an algorithmic change we are unable to make manual exceptions, but in cases of high quality content we can pass the examples along to the engineers who will look at them as they work on future iterations and improvements to the algorithm.”
Yes, it’s “ITAS.”
So what do YOU believe?…
Did a few high-profile sites make the whitelist, including a large site that recently completed an IPO and a smaller site whose owner is a former managing editor at WIRED? Is it fair that everyone else, whose guts are equally wrenched, whose lives are just as dislocated… is it fair that they anxiously wait for Google to improve the algorithm?
Whitelists and blacklists have great potential for good. Or they can be used purely to manage Google’s private agenda.
Who can trust Google now?
“Don’t be evil?”
All the best,
P.S. My next post will finally get to the 10 year case study. In the context of this post and its predecessor, it will be all the more meaningful.