How do content discovery engines, like Zemanta and Open Calais work?
<p>I was wondering how as semantic service like Open Calais figures out the names of companies, or people, tech concepts, keywords, etc. from a piece of text. Is it because they have a large database that they match the text against? </p>

<p>How would a service like Zemanta know what images to suggest to a piece of text for instance? </p>

<p>Michal Finkelstein from OpenCalais here.</p>

<p>First, thanks for your interest. I'll reply here but I also encourage you to read more on OpenCalais forums; there's a lot of information there including - but not limited to:
<a href="http://opencalais.com/tagging-information">http://opencalais.com/tagging-information</a>
<a href="http://opencalais.com/how-does-calais-learn">http://opencalais.com/how-does-calais-learn</a>
Also feel free to follow us on Twitter (@OpenCalais) or to email us at team@opencalais.com</p>

<p>Now to the answer:</p>

<p>OpenCalais is based on a decade of research and development in the fields of Natural Language Processing and Text Analytics.</p>

<p>We support the full "NLP Stack" (as we like to call it):
From text tokenization, morphological analysis and POS tagging, to shallow parsing and identifying nominal and verbal phrases.</p>

<p>Semantics come into play when we look for Entities (a.k.a. Entity Extraction, Named Entity Recognition). For that purpose we have a sophisticated rule-based system that combines discovery rules as well as lexicons/dictionaries. This combination allows us to identify names of companies/persons/films, etc., even if they don't exist in any available list.</p>

<p>For the most prominent entities (such as people, companies) we also perform anaphora resolution, cross-reference and name canonization/normalization at the article level, so we'll know that 'John Smith' and 'Mr. Smith', for example, are likely referring to the same person.
So the short answer to your question is - no, it's not just about matching against large databases.</p>

<p>Events/Facts are really interesting because they take our discovery rules one level deeper; we find relations between entities and label them with the appropriate type, for example M&amp;As (relations between two or more companies), Employment Changes (relations between companies and people), and so on. Needless to say, Event/Fact extraction is not possible for systems that are based solely on lexicons.
For the most part, our system is tuned to be precision-oriented, but we always try to keep a reasonable balance between accuracy and entirety.</p>

<p>By the way there are some cool new metadata capabilities coming out later this month so stay tuned.</p>

<p>Regards,</p>

<p>Michal</p>
