There using the Python Scikit-Learn library, which is one of the most popular machine learning libraries for Python.
Linear Regression Theory
The term "linearity" in algebra refers to a linear relationship between two or more variables. If we draw this relationship in a two dimensional space (between two variables, in this case), we get a straight line.
Let's consider a scenario where we want to determine the linear relationship between the numbers of hours a student studies and the percentage of marks that student scores in an exam. We want to find out that given the number of hours a student prepares for a test, about how high of a score can the student achieve? If we plot the independent variable (hours) on the x-axis and dependent variable (percentage) on the y-axis, linear regression gives us a straight line that best fits the data points, as shown in the figure below.
We know that the equation of a straight line is basically: and slope. There can be multiple straight lines depending upon the values of intercept and slope. Basically what the linear regression algorithm does is it fits multiple lines on the data points and returns the line that results in the least error.
This same concept can be extended to the cases where there are more than two variables. This is called multiple linear regression. For instance, consider a scenario where you have to predict the price of house based upon its area, number of bedrooms, average income of the people in the area, the age of the house, and so on. In this case the dependent variable is dependent upon several independent variables. A regression model involving multiple variables can be represented as:
y = b0 + m1b1 + m2b2 + m3b3 + ... ... mnbn
This is the equation of a hyper plane. Remember, a linear regression model in two dimensions is a straight line; in three dimensions it is a plane, and in more than three dimensions, a hyper plane.
Linear Regression with Python Scikit Learn
In this section we will see how the Python Scikit-Learn library for machine learning can be used to implement regression functions. We will start with simple linear regression involving two variables and then we will move towards linear regression involving multiple variables.
Simple Linear Regression
In this regression task we will predict the percentage of marks that a student is expected to score based upon the number of hours they studied. This is a simple linear regression task as it involves just two variables.
Importing Libraries
To import necessary libraries for this task, execute the following import statements:
import pandas as pd import numpy as np import matplotlib.pyplot as plt %matplotlib inline
Note: As you may have noticed from the above import statements, this code was executed using a Jupyter iPython Notebook.
Dataset
The dataset being used for this example has been made publicly available and can be downloaded from this link:
Note: This example was executed on a Windows based machine and the dataset was stored in "D:\datasets" folder. You can download the file in a different location as long as you change the dataset path accordingly.
The following command imports the CSV dataset using pandas:
dataset = pd.read_csv('D:\Datasets\student_scores.csv')
Now let's explore our dataset a bit. To do so, execute the following script:
dataset.shape
After doing this, you should see the following printed out:
(25, 2)
This means that our dataset has 25 rows and 2 columns. Let's take a look at what our dataset actually looks like. To do this, use the
head() method:
dataset.head()
The above method retrieves the first 5 records from our dataset, which will look like this:
To see statistical details of the dataset, we can use
describe():
dataset.describe()
And finally, let's plot our data points on 2-D graph to eyeball our dataset and see if we can manually find any relationship between the data. We can create the plot with the following script:
dataset.plot(x='Hours', y='Scores', style='o') plt.title('Hours vs Percentage') plt.xlabel('Hours Studied') plt.ylabel('Percentage Score') plt.show()
In the script above, we use
plot() function of the pandas dataframe and pass it the column names for
x coordinate and
y coordinate, which are "Hours" and "Scores" respectively.
The resulting plot will look like this:
From the graph above, we can clearly see that there is a positive linear relation between the number of hours studied and percentage of score.
Preparing the Data
Now we have an idea about statistical details of our data. The next step is to divide the data into "attributes" and "labels". Attributes are the independent variables while labels are dependent variables whose values are to be predicted. In our dataset we only have two columns. We want to predict the percentage score depending upon the hours studied. Therefore our attribute set will consist of the "Hours" column, and the label will be the "Score" column. To extract the attributes and labels, execute the following script:
X = dataset.iloc[:, :-1].values y = dataset.iloc[:, 1].values
The attributes are stored in the
X variable. We specified "-1" as the range for columns since we wanted our attribute set to contain all the columns except the last one, which is "Scores". Similarly the
y variable contains the labels. We specified 1 for the label column since the index for "Scores" column is 1. Remember, the column indexes start with 0, with 1 being the second column. In the next section, we will see a better way to specify columns for attributes and labels.
Now that we have our attributes and labels, the next step is to split this data into training and test sets. We'll do this by using Scikit-Learn's built-in
train_test_split() method:
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
The above script splits 80% of the data to training set while 20% of the data to test set. The
test_size variable is where we actually specify the proportion of test set.
Training the Algorithm
We have split our data into training and testing sets, and now is finally the time to train our algorithm. Execute following command:
from sklearn.linear_model import LinearRegression regressor = LinearRegression() regressor.fit(X_train, y_train)
With Scikit-Learn it is extremely straight forward to implement linear regression models, as all you really need to do is import the
LinearRegression class, instantiate it, and call the
fit() method along with our training data. This is about as simple as it gets when using a machine learning library to train on your data.
In the theory section we said that linear regression model basically finds the best value for the intercept and slope, which results in a line that best fits the data. To see the value of the intercept and slop calculated by the linear regression algorithm for our dataset, execute the following code.
To retrieve the intercept:
print(regressor.intercept_)
The resulting value you see should be approximately 2.01816004143.
For retrieving the slope (coefficient of x):
print(regressor.coef_)
The result should be approximately 9.91065648.
This means that for every one unit of change in hours studied, the change in the score is about 9.91%. Or in simpler words, if a student studies one hour more than they previously studied for an exam, they can expect to achieve an increase of 9.91% in the score achieved by the student previously.
Making Predictions
Now that we have trained our algorithm, it's time to make some predictions. To do so, we will use our test data and see how accurately our algorithm predicts the percentage score. To make pre-dictions on the test data, execute the following script:
y_pred = regressor.predict(X_test)
The
y_pred is a numpy array that contains all the predicted values for the input values in the
X_test series.
To compare the actual output values for
X_test with the predicted values, execute the following script:
df = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred}) df
The output looks like this:
Though our model is not very precise, the predicted percentages are close to the actual ones.
Note:
The values in the columns above may be different in your case because the
train_test_split function randomly splits data into train and test sets, and your splits are likely different from the one shown in this article.
Evaluating the Algorithm
The final step is to evaluate the performance of algorithm. This step is particularly important to compare how well different algorithms perform on a particular dataset. For regression algorithms, three evaluation metrics are commonly used:
Mean Absolute Error (MAE) is the mean of the absolute value of the errors. It is calculated as:
Mean Squared Error (MSE) is the mean of the squared errors and is calculated as:. Execute the following code: (but probably slightly different):
Mean Absolute Error: 4.183859899 Mean Squared Error: 21.5987693072 Root Mean Squared Error: 4.6474476121
You can see that the value of root mean squared error is 4.64, which is less than 10% of the mean value of the percentages of all the students i.e. 51.48. This means that our algorithm did a decent job.
Multiple Linear Regression
In the previous section we performed linear regression involving two variables. Almost all real world problems that you are going to encounter will have more than two variables. Linear regression involving multiple variables is called "multiple we will use multiple linear regression to predict the gas consumptions (in millions of gallons) in 48 US states based upon gas taxes (in cents), per capita income (dollars), paved highways (in miles) and the proportion of population that has a drivers license.
The details of the dataset can be found at this link:
The first two columns in the above dataset do not provide any useful information, therefore they have been removed from the dataset file. Now let's develop a regression model for this task.
Importing the Libraries
The following script imports the necessary libraries:
import pandas as pd import numpy as np import matplotlib.pyplot as plt %matplotlib inline
Dataset
The dataset for this example is available at:
The following command imports the dataset from the file you downloaded via the link above:
dataset = pd.read_csv('D:\Datasets\petrol_consumption.csv')
Just like last time, let's take a look at what our dataset actually looks like. Execute the
head() command:
dataset.head()
The first few lines of our dataset looks like this:
To see statistical details of the dataset, we'll use the
describe() command again:
dataset.describe()
Preparing the Data
The next step is to divide the data into attributes and labels as we did previously. However, unlike last time, this time around we are going to use column names for creating an attribute set and label. Execute the following script:
X = dataset[['Petrol_tax', 'Average_income', 'Paved_Highways', 'Population_Driver_licence(%)']] y = dataset['Petrol_Consumption']
Execute the following code to divide our data into training and test sets:
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
Training the Algorithm
And finally, to train the algorithm we execute the same code as before, using the
fit() method of the
LinearRegression class:
from sklearn.linear_model import LinearRegression regressor = LinearRegression() regressor.fit(X_train, y_train)
As said earlier,
The result should look something like this:
This means that for a unit increase in "petrol_tax", there is a decrease of 24.19 million gallons in gas consumption. Similarly, a unit increase in proportion of population with a drivers license results in an increase of 1.324 billion gallons of gas consumption. We can see that "Average_income" and "Paved_Highways" have a very little effect on the gas consumption.
Making Predictions
To make pre-dictions on the test data, execute the following script:
y_pred = regressor.predict(X_test)
To compare the actual output values for
X_test with the predicted values, execute the following script:
df = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred}) df
The output looks like this:
Evaluating the Algorithm
The final step is to evaluate the performance of algorithm. We'll do this by finding the values for MAE, MSE and RMSE. Execute the following script::
Mean Absolute Error: 45.8979842541 Mean Squared Error: 3609.37119141 Root Mean Squared Error: 60.0780425065
You can see that the value of root mean squared error is 60.07, which is slightly greater than 10% of the mean value of the gas consumption in all states. This means that our algorithm was not very accurate but can still make reasonably good predictions.
There are many factors that may have contributed to this inaccuracy, a few of which are listed here:
- Need more data: Only one year worth of data isn't that much, whereas having multiple years worth could have helped us improve the accuracy quite a bit.
- on of the most fundamental machine learning algorithms i.e. linear regression. We implemented both simple linear regression and multiple linear regression with the help of the Scikit-Learn machine learning library.
There are a few things you can do from here:
- Play around with the code and data in this article to see if you can improve the results (try changing the training/test size, transform/scale input features, etc.)
- Seek out some more complete resources on machine learning techniques, like the Deep Learning A-Z: Hands-On Artificial Neural Networks and Python for Data Science and Machine Learning Bootcamp courses, which will also teach you how to use Scikit-Learn and other popular ML libraries
- Download and play with some other regression datasets to see what kind of results you can get out of those. The more experience the better!
Have you used Scikit-Learn or linear regression on any problems in the past? If so, what was it and what were the results? Let us know in the comments!