and putting it into a form easily searchable by Impala and Hive. An explanation of how it is below. The script uses the Python CSV module to parse the traverse the file, row by row, and pick out the rows that require editing. It writes the new rows to a temporary file during execution and at the end replaces the source file with the temporary file. The thing the script will do to each row of the CSV is check how many columns it has. The standard tweet should have 14 rows. Many of the tweets in the shooting dataset had more than 14 rows and, originally, this threw the parser off. These extra rows were caused by commas in the text or source fields. To avoid this issue, we ignore any row with more than 14 columns. This results in some data loss but the amount lost is small; roughly 6% for the shooting dataset. If a tweet has the appropriate number of columns, the script then strips extraneous information from the column. This is done via regular expressions. There are different character source encodings depending on where the tweet originated from so both cases are handled. The column is then overwritten with the changes. source Next, the script will parse the day, month, and year from the column. This is done created_at via the Python dateutil and datetime modules. After parsing the values, and day, month, year columns are appended to the row. Finally, the script will strip the whitespace from the geo coordinate and time fields so that they can be properly imported into impala as doubles and integers respectfully. This script can be improved upon by instead of dropping malformated tweets, finding the cause of the improper formatting, repairing the situation, and correctly parsing the tweet. Lessons Learned Timeline/Schedule Feb 6 - Met with client to discuss project goals, assign project roles, setup project milestones
and putting it into a form easily searchable by Impala and Hive. An explanation of how it is below. The script uses the Python CSV module to parse the traverse the file, row by row, and pick out the rows that require editing. It writes the new rows to a temporary file during execution and at the end replaces the source file with the temporary file. The thing the script will do to each row of the CSV is check how many columns it has. The standard tweet should have 14 rows. Many of the tweets in the shooting dataset had more than 14 rows and, originally, this threw the parser off. These extra rows were caused by commas in the text or source fields. To avoid this issue, we ignore any row with more than 14 columns. This results in some data loss but the amount lost is small; roughly 6% for the shooting dataset. If a tweet has the appropriate number of columns, the script then strips extraneous information from the source column. This is done via regular expressions. There are different character encodings depending on where the tweet originated from so both cases are handled. The source column is then overwritten with the changes. Next, the script will parse the day, month, and year from the created_at column. This is done via the Python dateutil and datetime modules. After parsing the values, day, month, and year columns are appended to the row. Finally, the script will strip the whitespace from the geo coordinate and time fields so that they can be properly imported into impala as doubles and integers respectfully. This script can be improved upon by instead of dropping malformated tweets, finding the cause of the improper formatting, repairing the situation, and correctly parsing the tweet. Lessons Learned Timeline/Schedule Feb 6 - Met with client to discuss project goals, assign project roles, setup project milestones
DLRLCluster_Final_Report.docx