Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]
As a former Wikia employee, I am somewhat of a MediaWiki insider. I sped Wikia&#x27;s search engine up by several orders of magnitude and then went on to pilot a number of NLP&#x2F;machine learning initiatives in the company.<p>Jimmy Wales&#x27; already tried to make a &quot;Google Killer&quot; ten years ago. It was tilting at windmills to say the least. Letting individuals help manage algorithmic search results was harder than you could imagine. Let&#x27;s not even get into the difficulty of building an effective crawler.<p>One of Wikia&#x27;s former CEOs, Gil Penchina, notoriously undervalued search as a result of this very public gaffe. By the time I came in, it took over five seconds to do a simple on-wiki search. Searching across wikis took so long they actually just sent the search to Google and had you abandon the site. I personally fixed a lot of these problems, and that part was pretty cool.<p>So now let&#x27;s get to the subject at hand, which is a search feature based on an authoritative knowledge graph. Something like this should adequately surface factual information in an intuitive manner -- optimally based on natural language. Wikia already tried this, too. They brought on a very seasoned advisor who played a crucial role in the semantic web movement far back into the early oughts. I remember going to semantic web meetups in Austin when I was in grad school quite some time ago now to hear this guy talk.<p>This guy was essentially the SF-based manager or lead for a small team located in Poland whose job it was to take some of the &quot;structured data&quot; at Wikia and attempt to build some kind of knowledge graph on top of it. This project was unsuccessful.<p>So why did it fail? We&#x27;ll start with a lack of product direction. Wikia had and probably still has a very junior product organization that is mostly interested in the site&#x27;s UI and (recently) a focus on &quot;fandom&quot; (yuck). The team allocated to the project was based in Poland (Poznan, to be exact), and primarily kids coming out of a technical school on their first job. Your assumption about communication being a problem would be correct. However, the subject matter expert was so entrenched in his area of specialization, the problem was even more compounded on the native English-speaker side. There was too much getting in the weeds, and not enough focus on incremental progress.<p>To make things worse, they tried using a proprietary, not-ready-for-primetime data store because it most closely matched the SME&#x27;s preconceptions on how the data should be structured. There was absolutely not an existing business use case for this data store, and problems getting it to work turned even building a simple demo into a death march.<p>Either way, what I&#x27;m saying is, $250,000 is not enough to solve this problem. We have attempted to solve this problem before in the MediaWiki world. It&#x27;s not going to magically get better. To make something like this work, you need:<p>1) Best-in-class UX people who would know how a knowledge graph provides a significant improvement over existing solutions
2) Leadership that can bridge the gap between SMEs and implementers
3) Very skilled engineering resources with backgrounds in less conventional technologies<p>This is a massive investment that no one is willing to spend on what is essentially a media play.<p>About six months later, I had built a proof-of-concept that sucked data out of MediaWiki Infobox templates into Neo4j, a well supported graph database. I was able to answer questions like, &quot;Which cartoon characters are rabbits&quot;, and &quot;What movie won the most Oscars in 1968&quot; using the Cypher query language.<p>At that point in time, Wikia had decided they were tired of investing in structured data, and wanted to re-skin the site for a third time in as many years to make it look more like BuzzFeed.<p>Structured data is cool. In many cases, unsupervised learning may be what you&#x27;re actually looking for. But in the end it has to satisfy a real user&#x27;s needs.<p>Wikipedia has five million English articles. Wikia has over 20 million. As far as capitalizing on this wealth of knowledge, the devil is truly in the details. But it&#x27;s a real shame that all of that information isn&#x27;t put to better use than to encourage the socially maladjusted to take quizzes over which anime character they&#x27;re more like.