Podcast: Play in new window | Download

My relationship with Google News has always run hot and cold. No make that cold and tepid. From the very beginning of Google News as an experiment back in 2001, they refused to index my work, which they said was my fault, not theirs (“they” being an algorithm attached to an e-mail box, of course). But new evidence has recently come to light suggesting to me that Google News has an actual blacklist.

For those not familiar with the expression, “blacklist” usually refers to Hollywood screen and television writers from the 1950’s McCarthy era who were thought to be communist sympathizers and were banned from working openly in the entertainment industry as a result. Not Hollywood’s finest hour.  My suspicion is that Google News has a similar list of writers it would prefer did not exist and these people (including me) are systematically excluded from having their work indexed and publicized.

Despite having given Google some of its earliest publicity in the form of the company’s first-ever TV interviews back in 1998, I’m told by friends inside the Googleplex that I have enemies in high places there, which I find flattering.

Google’s excuse for excluding me has always been that I am a sole proprietor of this operation and therefore what I produce doesn’t qualify as news. Tell that to the hundreds of companies I’ve been the first to find, the dozens of big stories I have been the first (and sometimes ONLY) reporter to cover, including a bunch of big stories about Google.

Nothing personal, it’s just the algorithm, they say.

We’re not evil, we’re just programmed that way.

Yeah, right.

When I was at PBS I pointed out to Google News that my URL was pbs.org/cringely, so I was really part of a larger operation that included news sources Google was indexing. If they indexed the NewsHour, I argued, then they had to index me. And so they eventually did — for a few weeks at a time. But I’d inevitably fall off the Google index again while some other bozo, often with a tenth or fewer readers than me, would stay on.

When I wasn’t being indexed by Google (which remains the case today) the only way my work would appear in Google News was when some other writer would cite me, which fortunately happens most weeks. Still, such second generation coverage doesn’t bring me any real traffic, I’m pretty sure.

Here’s how it would go down.  I’d fall off Google News, a PBS lawyer would write to Google and I’d eventually be back on for a few days until it started all over again. The times I wasn’t indexed lasted for weeks. And when I left PBS and arrived here, well that was it. Now I really was a one-man band.

So I was very excited a couple months ago when I began writing for AOL’s Housingwatch.com site, where I currently file 1-2 pieces per week on real estate and mortgages (here’s the short version: the mortgage industry is screwed, film at 11). With something like 30 professional writers working at Housingwatch alone, I knew Google would have no choice but to index my work.

Nope, I guessed wrong.

To my surprise it looks like none of the coverage at Housingwatch or its parent AOL is being indexed at all, which is stupid considering AOL serves 15+ million readers per day.

A week ago I wrote a short piece for Housingwatch that was picked-up by the AOL home page and given prominent play. As a result it got more than 750 reader comments over one weekend — a huge number.

If you want to figure how many people read a column from the number of comments, try multiplying by at least 1000. My posts here tend to average over 100 comments, which is pretty good.  But 750 comments, that’s spectacular.

Yet when that particular AOL column was burning-up the Internet I saw nothing about it on Google News. It was invisible.

How could that be? Most of the top stories on Google News weren’t getting 750,000+ readers, I was sure, yet they were being indexed.

My theory, then, is that both AOL and I are blacklisted from Google News.  That’s hundreds of  professional writers… invisible.

AOL is in many ways a Google competitor, I suppose. Certainly it is run by an ex-Googler, who probably left behind bad feelings.  But, if true, how fair is this on Google’s part? Not very. I don’t suppose it qualifies as evil, but it is definitely petty and in no way serves Google users, who should be appalled.

Podcast: Play in new window | Download

Rupert Murdoch said recently that he’s planning to stop Google News from indexing his publications including the Times of London and the Wall Street Journal. Murdoch’s idea is that Google News and the like make it too easy for Internet users to sample news for free rather than paying for it as God and Rupert intended. Mark Cuban, who is very clever but with whom I rarely agree, thinks this is smart on Murdoch’s part, because Twitter is changing the way people find news, effectively disintermediating Google, but not the News Corp. publications, themselves.

It’s funny how Murdoch’s statement made Cuban think of Twitter while it made me think immediately of the A&P.

The Great Atlantic and Pacific Tea Company, or A&P, was America’s first national chain of food markets. Hell, it was America’s first self-serve market, first to have store brands, first to advertise nationally, first to have a customer loyalty program (in 1912!), first to publish its own magazine (Womens’ Day, which is still around, though no longer owned by the A&P), and for most of my childhood back in Ohio A&P was the big Kahuna of grocery chains. With $5.4 billion in sales in the mid-1960s, A&P was at least 20 percent bigger than any of its competitors.

But after 105 years of setting the pace for the grocery industry, A&P peaked in the mid-1960s and went into a decline that lasted for at least 15 years and, it can be argued, continues even to this day. A&P, which has had German owners (the Tengelman Group) since the 1970s, is more of a super-regional chain today and doesn’t particularly vie for industry leadership on any measure. What happened in the mid-1960s to hurt A&P was it opted out of being indexed by Google News.

Well not literally, but close enough. A&P management, which back in the mid-60′s was still chosen from the founding Hartford family, decided at that time to abandon shopping centers — retail aggregators as Google is a news aggregator. They reasoned that in most shopping centers the anchor store was an A&P. In their view their supermarket was the main draw for a shopping center and didn’t need any of those other shops or stores to provide traffic. The rest of the shopping center was seen by A&P management as being purely parasitic. The company could get cheaper real estate down the road with a standalone store, which is why today most A&Ps aren’t in shopping centers. It’s also why A&P is a shadow of its former self.

You see the Hartford family (and Rupert Murdoch) were wrong. The flawed assumption at A&P was that shopping centers would somehow do without an anchor supermarket, which they didn’t. By withdrawing from the common location A&P was not only walking away from significant customer traffic, it was in each case simply handing that traffic to a Safeway or a Kroger store. It was a supremely stupid move.

Which brings us back to Rupert Murdoch, who is brilliant in his own right but in this case can’t find his own URL with both hands.  If Murdoch abandons Google News, then those hundreds of millions of reader referrals per day will simply go to other publications or maybe even to guys like me.  It’s not like Google can’t fill the space.

Murdoch wants readers to pay for news. I’d like folks to be paying for my words, too. But pulling out of Google News isn’t the way for either of us to accomplish that. And Twitter isn’t a factor with enough of the audience (yet? ever?) to make a difference.

Giving Murdoch the benefit of the doubt, then, I’m guessing he simply doesn’t mean what he said.  Perhaps he just wanted to sow a little confusion, get some publicity and maybe a concession or two from Google.

It won’t work.