Last year I wrote about how Search, as a technology, is dead. I explained the problem with search results pages(it’s not ads), Google’s technological advantage, and the incentives that stop Google and other search engines from improve.
The year before, I wrote about how Neeva is not really a solution to Google. Back then Neeva was just a press release. They have since tweaked their model a little. However my original point still stands; Neeva is just trying to sell people Bing, at a premium. They neither have the means, nor the intent to create a technological rival to Google. I posited, and still hold this opinion, that Neeva is just market validation for a potential Apple M&A exit play.
Why am I talking about this again? Because this issue recently went viral on Twitter and Hacker News. YCombinator’s Michael Seibel made a thread about how Google results have gone downhill in the categories of Health, Products reviews and Recipes. The replies and comments made me think about this issue in another aspect.
Last time, I attacked the issue from the supply side. This time I will talk about the demand side; the open web. Which is in a much more dire condition than the search engines.
Running off cliffs
Humanity is very adaptable. Given any new constraint on our situation, we would reconfigure our whole life around it and still thrive. This applies physically and also mentally. This also applies to the built world. We make buildings, cars, infrastructures, etc. around the economical and political constraints. For example, you can figure out what era any given London building is based on the dimensions of the bricks used. The brick size was based on the the constraints of their respective Tax policy.
Same goes of websites. The constraints on any web based business are traffic sources; Facebook and Google have cornered the market for most of the world. Whether a business is engaged in commerce(selling and buying stuff), or a media business, which means you need traffic for the attention money.
So you will adapt your site or app to the Facebook’s and Google’s algorithms. Facebook’s algorithm rewards engagement. And the best way to achieve that is to have inflammatory headlines. Google’s algorithm is looking for semantic-tags, a specific English writing style and backlinks. This gives rise to blogspam and affiliate farming listicles. Product reviews and product lists have very little utility left for the average user.
I also wrote about how to build an egalitarian search-engine for an open web. However, if we were to build a holistic search engine that gives us the best results for any given query, it would not be enough. It would not have any quality sites to index. The web has been transformed by the garbage processing machines that are these algorithms.
What can a paid subscription search service achieve in this regard? They can stop tracking you. They can stop serving you ads. They can’t turn back the clock on the open web. The ship has sailed.
Niches
Paul Graham chimed in to point out how disruption happens through niches.
Recipes already have aggregators. These are the sites you see as the top organic result. The recipes are surrounded by useless fluff. The point is to make you scroll, which creates more ad impressions, which means more revenue. These incentives can’t be changed no matter how much you pay to search “duck l’orange”. Furthermore, all niche aggregators and content farms like AllRecipes, Epicurious, WebMD, etc. can double as search engine as well. A media business thrives on quantity, not quality.
Business Reality
You can’t make a niche search engine because the business reality does not make sense. The technical requirements to build a search engine that only looks for “good” recipe web pages are the same as rebuilding Google. The other solution is to create yet another aggregator for the niche. An aggregator that is driven by the same incentives that all of the others that are the cause(or a symptom) of the problem.
A caveat: You can spend a lot of money on marketing. Creating market conditions that define differentiation. Even if you are successful, I doubt the market is big enough to justify to cost to create the ‘Apple’ of recipe aggregators. The business case does not exist. You may find a VC to fund this project by showing him a dream acquisition by Conde Nast or some other big media conglomerate, but I doubt he’ll(lets be honest, it’ll be a he) be at his job long enough for you to reach that exit.
The Solution
The solution to the recipe problems is simple; buy cook books. There are many of them, they are made by professionals and they do not have any fluff around them. The solution to the medical advice problem is visiting a doctor, or any qualified healthcare provider. The best way to solve the product review problem is to buy from a vendor that has a generous return policy. And to buy just the stuff you need. Many will tell you to try reddit. Hate to burst their bubble, but reddit is also full of sock puppets, spam bots and astroturfers.
As for the search engine question; I layed-out certain principles, and their downsides to build an egalitarian, decentralized search engine. However, for it to work we need the web to exist as it did before it was SEO’d to death. I do not see that happening any time soon.