Nerd Industries: Stuart Robinson's blog

It's been a while (roughly 6 months) since I last posted something on this blog. In fact, I've been so remiss that I made one of my new year's resolutions is to try to post a bit more regularly. Towards that end, I thought I'd comment on an op-ed about Google that recently appeared in the New York Times: "Search, but You May Not Find".

It's written by Adam Raff, who apparently has an axe to grind with Google because his company (Foundem) developed a vertical search engine that didn't show up highly enough in Google search results to be sufficiently visible to compete with Google. I'm not making this up. Here are his own words:

"One way that Google exploits this control is by imposing covert “penalties” that can strike legitimate and useful Web sites, removing them entirely from its search results or placing them so far down the rankings that they will in all likelihood never be found. For three years, my company’s vertical search and price-comparison site, Foundem, was effectively “disappeared” from the Internet in this way."

It beggars belief that this op-ed piece was even published, given its relentless self-advertising based on unverifiable claims--e.g., "Google’s treatment of Foundem stifled our growth and constrained the development of our innovative search technology." The inherent bias in the article is pretty hard to miss.

But let's ignore Raff's obvious bias for a moment and consider his argument on its merits. His argument is basically that net neutrality doesn't go far enough in regulating the internet. Search engines should be regulated as much as internet service providers are, because both are the gateway to the web.

The flaws in his argument are legion. The biggest one is the unanalyzed assumption that "net neutrality" is a good thing. (That's hotly debated.) He doesn't even say what he understand the term to mean (it's not self-explantory) or argue for its merits. Then there's the lumping together or internet service providers and search engines. That's also a whopper. (Without the former, you're locked out of the internet. Without the latter, you'll just have a hard time finding what you're looking for.) And then there's the failure to recognize that there's an alternative to regulation--namely, healthy competition. And that's not going to come from government regulation. (I'm not a Libertarian, and don't believe in the unfettered free market, but that doesn't mean that I don't see the merits of competition over regulation.)

I expect better from The New York Times. (Apparently, I'm not alone, judging from other blog responses.)

Last night Powerset hosted the SF Semantic Web Meetup, organized by Marco Neumann, where we gave a talk about NLP and the Semantic Web. I presented some work in progress on instant answers (slides) and my colleague Scott Waterman presented ongoing work by our group (Text Processing for Semantic Applications) on triples extraction using our natural language pipeline. In addition, Bill Flitter from Dlvr.it talked about the publishing industry in the emerging world of the realtime web.

We enjoyed giving our presentations and appreciated the feedback that we received. One of the questions that came up during the question time after my talk was one that I frequently encounter, which is why finite state transducers aren't just regular expressions. What I normally say, but perhaps failed to say clearly enough on this occasion, is that finite state transducers are regular expression, but they do a lot more. As my former colleague Brendan O'Connor used to say, "Finite state transducers are regular expressions on steroids!" I think that's right, and at some point I should write up a more detailed and technical explanation of what that means.