Table of Contents Technical Manual i Version History ii Table of Contents iii of Figures 1 1.Introduction 2 2. Installation Guideline 2 2.1 Prerequisite Software 2 2.2 Software Installation 2 2.2.1 Scrapy 2 2.2.2 Facebook-SDK 2 2.2.3 Psycopg2 2 2.2.4 Python 2.7 2 2.2.5 pip 2 2.2.6 PostgreSQL database 2 2.2.7 Ruby on Rails 3 2.2.8 Tweepy 3 3. Modules 3 3.1 Crawler 3 3.1.1 Scrapy 3 3.1.1.1 Structure 3 3.1.1.2 3 3.1.1.3 spiderName.py 4 3.1.1.4 Pipelines.py 8 3.1.2 BeautifulSoup 8 3.1.2.1 Program Flow 8 3.1.2.2 Implementation 9 3.2 Facebook 10 3.2.1Structure 10 3.2.1.1 Classes 10 3.2.2 Workflow 14 3.2.2.1 General Program Flow 14 3.2.2.2 postsParser.parse workflow 15 3.2.3 Q & A 16 3.3 Twitter 16 3.3.1.Structure 17 3.3.2 Workflow 22 3.3.2.1 General Program Flow 22 3.3.2.2 Get Function workflow 23 3.3.3 Q & A 23 3. 4 Data Ingestion 24 3.4.1 Connect to Database 25 3.4.2 Create custom dictionary 26 3.4.3 Identify duplicates 27 3.4.4 Insert into Database 28 3.4.5.1 Tasks Tab 29 3.5 Scheduler 30
Table of Contents Technical Manual i Version History ii Table of Contents iii of Figures 1 1.Introduction 2 2. Installation Guideline 2 2.1 Prerequisite Software 2 2.2 Software Installation 2 2.2.1 Scrapy 2 2.2.2 Facebook-SDK 2 2.2.3 Psycopg2 2 2.2.4 Python 2.7 2 2.2.5 pip 2 2.2.6 PostgreSQL database 2 2.2.7 Ruby on Rails 3 2.2.8 Tweepy 3 3. Modules 3 3.1 Crawler 3 3.1.1 Scrapy 3 3.1.1.1 Structure 3 3.1.1.2 3 3.1.1.3 spiderName.py 4 3.1.1.4 Pipelines.py 8 3.1.2 BeautifulSoup 8 3.1.2.1 Program Flow 8 3.1.2.2 Implementation 9 3.2 Facebook 10 3.2.1Structure 10 3.2.1.1 Classes 10 3.2.2 Workflow 14 3.2.2.1 General Program Flow 14 3.2.2.2 postsParser.parse workflow 15 3.2.3 Q & A 16 3.3 Twitter 16 3.3.1.Structure 17 3.3.2 Workflow 22 3.3.2.1 General Program Flow 22 3.3.2.2 Get Function workflow 23 3.3.3 Q & A 23 3. 4 Data Ingestion 24 3.4.1 Connect to Database 25 3.4.2 Create custom dictionary 26 3.4.3 Identify duplicates 27 3.4.4 Insert into Database 28 3.4.5.1 Tasks Tab 29 3.5 Scheduler 30