5.2 Data Description The data has multiple different properties; see examples in Table. 5.2.1. Columns represent variables, while the values are measurements of these variables for collected samples. Table 5.2.1 Sample Data date discharge stage DO_pct DO pH spCond temp. turbidity 3/23/2013 0.120017 0.156 77.3 9.77 7.61 0.719 5.34 8.2 2:30 3/23/2013 0.118492 0.155 2:40 3/23/2013 77.4 9.78 7.61 0.719 5.36 9.2 2:45 3/23/2013 0.116976 0.154 2:50 3/23/2013 0.120017 0.156 77.6 9.78 7.61 0.719 5.39 8.8 3:00 3/23/2013 0.118492 0.155 3:10 3/23/2013 77.6 9.79 7.61 0.719 5.4 8.2 3:15 As shown in Figure 5.2.1, some data records do not have a value for every single property, which may be due to the rate at which the data was being gathered. This is another factor that could affect how we should manage these incomplete records when we implement the database. We could solve this issue by merging an incomplete row with the row before or after it, since it seems to be an issue of timing. 5.3 Possible Design Solutions 5.3.1 MySQL Function Since the data we currently have is in CSV format, it would be ideal if we could easily import all of it into a MySQL table. One of the team members has done exactly this in the past with a built-in SQL command. This command allows us to format the table just like it is formatted in the CSV, then directly inject all the data into a new or pre existing MySQL table. If we use this and it is successful, it will solve the issue. For normalization, we will look into splitting the data into several tables.
5.2 Data Description The data has multiple different properties; see examples in Table. 5.2.1. Columns represent variables, while the values are measurements of these variables for collected samples. Table 5.2.1 Sample Data date discharge stage DO_pct DO pH spCond temp. turbidity 3/23/2013 2:30 0.120017 0.156 77.3 9.77 7.61 0.719 5.34 8.2 3/23/2013 2:40 0.118492 0.155 3/23/2013 2:45 77.4 9.78 7.61 0.719 5.36 9.2 3/23/2013 2:50 0.116976 0.154 3/23/2013 3:00 0.120017 0.156 77.6 9.78 7.61 0.719 5.39 8.8 3/23/2013 3:10 0.118492 0.155 3/23/2013 3:15 77.6 9.79 7.61 0.719 5.4 8.2 As shown in Figure 5.2.1, some data records do not have a value for every single property, which may be due to the rate at which the data was being gathered. This is another factor that could affect how we should manage these incomplete records when we implement the database. We could solve this issue by merging an incomplete row with the row before or after it, since it seems to be an issue of timing. 5.3 Possible Design Solutions 5.3.1 MySQL Function Since the data we currently have is in CSV format, it would be ideal if we could easily import all of it into a MySQL table. One of the team members has done exactly this in the past with a built-in SQL command. This command allows us to format the table just like it is formatted in the CSV, then directly inject all the data into a new or pre existing MySQL table. If we use this and it is successful, it will solve the issue. For normalization, we will look into splitting the data into several tables.
StreamFieldFinalReport.docx