1. Introduction The focus of our team is developing the topic analysis component of the information retrieval system. Topic analysis helps in discovering abstract topics from tweets and webpages. This would enable the search and retrieval of documents based on topics. The frontend team uses the topics that are identified by our module to design a faceted search. The topics that are identified can also be used in validating results of the clustering team. For designing a topic model we will use the Latent Dirichlet Allocation (LDA) [1] technique. The overall report has been subdivided into various sections. First, a literature review and related references are provided for readers to understand the algorithm of LDA and if required, to know details about the LDA model. Second, we give the overview of our work plan. From this part, readers will get an overall idea of our work outline to understand the function of LDA in the whole project and know the relations and inter-dependencies between our team and other teams. Then we go on to describe in detail the overview of the whole information retrieval system framework and describe the inter-dependencies between different teams. Sections 4.1 4.4 describe in detail the implementation and evaluation of our approach. Finally, we include the user’s manual that contains a step-by-step walk-through of the important software packages and relevant software installation. 2. Background Knowledge and Literature Review 2.1 Important Chapters for Topic Analysis The textbook for the course ‘Introduction to Information Retrieval’ by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze [3] has been an indispensable source of information for gaining background knowledge about topic modeling and putting it in perspective with respect to other teams. The chapters relevant for topic modeling are chapter 6, 7, 8 and 18. Among them, chapters 6 and 7 are the most important chapters. Chapter 6, which is “Scoring, term weighting and
1. Introduction The focus of our team is developing the topic analysis component of the information retrieval system. Topic analysis helps in discovering abstract topics from tweets and webpages. This would enable the search and retrieval of documents based on topics. The frontend team uses the topics that are identified by our module to design a faceted search. The topics that are identified can also be used in validating results of the clustering team. For designing a topic model we will use the Latent Dirichlet Allocation (LDA) [1] technique. The overall report has been subdivided into various sections. First, a literature review and related references are provided for readers to understand the algorithm of LDA and if required, to know details about the LDA model. Second, we give the overview of our work plan. From this part, readers will get an overall idea of our work outline to understand the function of LDA in the whole project and know the relations and inter-dependencies between our team and other teams. Then we go on to describe in detail the overview of the whole information retrieval system framework and describe the inter-dependencies between different teams. Sections 4.1 4.4 describe in detail the implementation and evaluation of our approach. Finally, we include the user’s manual that contains a step-by-step walk-through of the important software packages and relevant software installation. 2. Background Knowledge and Literature Review 2.1 Important Chapters for Topic Analysis The textbook for the course ‘Introduction to Information Retrieval’ by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze [3] has been an indispensable source of information for gaining background knowledge about topic modeling and putting it in perspective with respect to other teams. The chapters relevant for topic modeling are chapter 6, 7, 8 and 18. Among them, chapters 6 and 7 are the most important chapters. Chapter 6, which is “Scoring, term weighting and
FinalReport-TopicAnalysis.docx