

# Data manipulation using pandas: importing, cleaning, and preprocessing data



To begin, let's import the pandas library and create a simple dataset.

```python
import pandas as pd

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000]
}

df = pd.DataFrame(data)
```

Now that we have a dataset, let's clean and preprocess it. One common task is to handle missing values.

```python
# Add a row with missing values
data['age'].append(None)
data['income'].append(75000)

df = pd.DataFrame(data)
```

To handle missing values, we can use the `dropna()` function to remove rows with missing values.

```python
df_cleaned = df.dropna()
```

Another common task is to convert categorical variables to numerical variables.

```python
# Add a column with categorical data
data['gender'] = ['M', 'F', 'M', 'F', 'M']

df = pd.DataFrame(data)
```

We can use the `pd.get_dummies()` function to convert the categorical column to a numerical one.

```python
df_preprocessed = pd.get_dummies(df, columns=['gender'])
```

## Exercise
1. Import the pandas library.
2. Create a dataset with missing values and categorical variables.
3. Clean the dataset by removing rows with missing values.
4. Preprocess the dataset by converting the categorical column to a numerical one.

### Solution
```python
import pandas as pd

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)

df_cleaned = df.dropna()

df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])
```

# Linear regression: a simple inversion problem

Linear regression is a simple statistical method used to model the relationship between a dependent variable and one or more independent variables. It is a type of inversion problem because it involves finding the best-fit line that minimizes the sum of the squared differences between the observed data points and the predicted values.

The equation for a simple linear regression is:

$$y = \beta_0 + \beta_1 * x$$

where $y$ is the dependent variable, $x$ is the independent variable, $\beta_0$ is the intercept, and $\beta_1$ is the slope.

To perform linear regression using scikit-learn, we will use the `LinearRegression` class.

```python
from sklearn.linear_model import LinearRegression

X = df_preprocessed[['age']]
y = df_preprocessed['income']

model = LinearRegression()
model.fit(X, y)
```

## Exercise
1. Import the pandas library and create a dataset with a dependent variable and an independent variable.
2. Clean the dataset by removing rows with missing values.
3. Preprocess the dataset by converting the categorical column to a numerical one.
4. Perform linear regression using scikit-learn.

### Solution
```python
import pandas as pd
from sklearn.linear_model import LinearRegression

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)

df_cleaned = df.dropna()

df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])

X = df_preprocessed[['age']]
y = df_preprocessed['income']

model = LinearRegression()
model.fit(X, y)
```

# Inversion in linear regression using scikit-learn



First, let's import the necessary libraries and create a dataset.

```python
import pandas as pd
from sklearn.linear_model import LinearRegression

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)
```

Next, we will clean and preprocess the dataset.

```python
df_cleaned = df.dropna()
df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])
```

Now, we can perform linear regression using scikit-learn.

```python
X = df_preprocessed[['age']]
y = df_preprocessed['income']

model = LinearRegression()
model.fit(X, y)
```

## Exercise
1. Import the pandas library and create a dataset with a dependent variable and an independent variable.
2. Clean the dataset by removing rows with missing values.
3. Preprocess the dataset by converting the categorical column to a numerical one.
4. Perform linear regression using scikit-learn.

### Solution
```python
import pandas as pd
from sklearn.linear_model import LinearRegression

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)

df_cleaned = df.dropna()
df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])

X = df_preprocessed[['age']]
y = df_preprocessed['income']

model = LinearRegression()
model.fit(X, y)
```

# Nonlinear regression: solving for the best-fit curve

Nonlinear regression is a type of inversion problem that involves finding the best-fit curve that minimizes the sum of the squared differences between the observed data points and the predicted values. The curve is nonlinear because it is not a straight line.

The equation for a nonlinear regression is:

$$y = \beta_0 + \beta_1 * x + \beta_2 * x^2$$

where $y$ is the dependent variable, $x$ is the independent variable, $\beta_0$ is the intercept, $\beta_1$ is the slope, and $\beta_2$ is the curvature.

To perform nonlinear regression using scikit-learn, we will use the `PolynomialFeatures` class to create a nonlinear feature matrix and the `LinearRegression` class to perform linear regression on the nonlinear feature matrix.

```python
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

X = df_preprocessed[['age']]
y = df_preprocessed['income']

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

model = LinearRegression()
model.fit(X_poly, y)
```

## Exercise
1. Import the necessary libraries and create a dataset.
2. Clean the dataset by removing rows with missing values.
3. Preprocess the dataset by converting the categorical column to a numerical one.
4. Perform nonlinear regression using scikit-learn.

### Solution
```python
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)

df_cleaned = df.dropna()
df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])

X = df_preprocessed[['age']]
y = df_preprocessed['income']

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

model = LinearRegression()
model.fit(X_poly, y)
```

# Inversion in nonlinear regression using scikit-learn



First, let's import the necessary libraries and create a dataset.

```python
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)
```

Next, we will clean and preprocess the dataset.

```python
df_cleaned = df.dropna()
df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])
```

Now, we can perform nonlinear regression using scikit-learn.

```python
X = df_preprocessed[['age']]
y = df_preprocessed['income']

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

model = LinearRegression()
model.fit(X_poly, y)
```

## Exercise
1. Import the necessary libraries and create a dataset.
2. Clean the dataset by removing rows with missing values.
3. Preprocess the dataset by converting the categorical column to a numerical one.
4. Perform nonlinear regression using scikit-learn.

### Solution
```python
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

data = {
    'age': [25, 30, 35, 40, 45],
    'income': [50000, 60000, 70000, 80000, 90000],
    'gender': ['M', 'F', 'M', 'F', 'M']
}

df = pd.DataFrame(data)

df_cleaned = df.dropna()
df_preprocessed = pd.get_dummies(df_cleaned, columns=['gender'])

X = df_preprocessed[['age']]
y = df_preprocessed['income']

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

model = LinearRegression()
model.fit(X_poly, y)
```

# Model evaluation: understanding and interpreting metrics

Model evaluation is an important step in machine learning to understand how well a model is performing and to make informed decisions about its use.

There are several metrics used to evaluate machine learning models, such as accuracy, precision, recall, and F1 score.

To calculate these metrics, we can use the `accuracy_score`, `precision_score`, `recall_score`, and `f1_score` functions from the scikit-learn library.

```python
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 1, 2]

accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred, average='macro')
recall = recall_score(y_true, y_pred, average='macro')
f1 = f1_score(y_true, y_pred, average='macro')
```

## Exercise
1. Import the necessary functions from the scikit-learn library.
2. Create true and predicted values.
3. Calculate the accuracy, precision, recall, and F1 score.

### Solution
```python
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 1, 2]

accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred, average='macro')
recall = recall_score(y_true, y_pred, average='macro')
f1 = f1_score(y_true, y_pred, average='macro')
```

# Real-world example: predicting house prices using inversion



First, let's import the necessary libraries and load the dataset.

```python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

data = pd.read_csv('house_sales.csv')
```

Next, we will clean and preprocess the dataset.

```python
data_cleaned = data.dropna()
data_preprocessed = pd.get_dummies(data_cleaned, columns=['location'])
```

Now, we can split the dataset into training and testing sets.

```python
X = data_preprocessed[['size', 'location_A', 'location_B']]
y = data_preprocessed['price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

Finally, we can perform linear regression using scikit-learn and evaluate the model.

```python
model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
```

## Exercise
1. Import the necessary libraries and load the dataset.
2. Clean and preprocess the dataset.
3. Split the dataset into training and testing sets.
4. Perform linear regression using scikit-learn.
5. Evaluate the model using the mean squared error metric.

### Solution
```python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

data = pd.read_csv('house_sales.csv')

data_cleaned = data.dropna()
data_preprocessed = pd.get_dummies(data_cleaned, columns=['location'])

X = data_preprocessed[['size', 'location_A', 'location_B']]
y = data_preprocessed['price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
```

# Real-world example: image restoration using inversion



First, let's import the necessary libraries and load the dataset.

```python
import numpy as np
import cv2
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

image = cv2.imread('noisy_image.jpg', cv2.IMREAD_GRAYSCALE)
noise = cv2.randn(image.shape, 0, 10)
noisy_image = image + noise
```

Next, we will perform inversion using scikit-learn.

```python
X = np.array(noisy_image).reshape(-1, 1)
y = np.array(image).reshape(-1, 1)

model = LinearRegression()
model.fit(X, y)

restored_image = model.predict(X)
mse = mean_squared_error(y, restored_image)
```

## Exercise
1. Import the necessary libraries and load the dataset.
2. Perform inversion using scikit-learn.
3. Evaluate the model using the mean squared error metric.

### Solution
```python
import numpy as np
import cv2
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

image = cv2.imread('noisy_image.jpg', cv2.IMREAD_GRAYSCALE)
noise = cv2.randn(image.shape, 0, 10)
noisy_image = image + noise

X = np.array(noisy_image).reshape(-1, 1)
y = np.array(image).reshape(-1, 1)

model = LinearRegression()
model.fit(X, y)

restored_image = model.predict(X)
mse = mean_squared_error(y, restored_image)
```

# Real-world example: predicting customer churn using inversion



First, let's import the necessary libraries and load the dataset.

```python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import accuracy_score

data = pd.read_csv('customer_data.csv')
```

Next, we will clean and preprocess the dataset.

```python
data_cleaned = data.dropna()
data_preprocessed = pd.get_dummies(data_cleaned, columns=['segment', 'country'])
```

Now, we can split the dataset into training and testing sets.

```python
X = data_preprocessed[['age', 'segment_A', 'segment_B', 'country_A', 'country_B']]
y = data_preprocessed['churn']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

Finally, we can perform linear regression using scikit-learn and evaluate the model.

```python
model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
```

## Exercise
1. Import the necessary libraries and load the dataset.
2. Clean and preprocess the dataset.
3. Split the dataset into training and testing sets.
4. Perform linear regression using scikit-learn.
5. Evaluate the model using the accuracy metric.

### Solution
```python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import accuracy_score

data = pd.read_csv('customer_data.csv')

data_cleaned = data.dropna()
data_preprocessed = pd.get_dummies(data_cleaned, columns=['segment', 'country'])

X = data_preprocessed[['age', 'segment_A', 'segment_B', 'country_A', 'country_B']]
y = data_preprocessed['churn']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
```

# Real-world example: automated trading using inversion



First, let's import the necessary libraries and load the dataset.

```python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

data = pd.read_csv('stock_prices.csv')
```

Next, we will clean and preprocess the dataset.

```python
data_cleaned = data.dropna()
data_preprocessed = pd.get_dummies(data_cleaned, columns=['company'])
```

Now, we can split the dataset into training and testing sets.

```python
X = data_preprocessed[['date', 'company_A', 'company_B']]
y = data_preprocessed['price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

Finally, we can perform linear regression using scikit-learn and evaluate the model.

```python
model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
```

## Exercise
1. Import the necessary libraries and load the dataset.
2. Clean and preprocess the dataset.
3. Split the dataset into training and testing sets.
4. Perform linear regression using scikit-learn.
5. Evaluate the model using the mean squared error metric.

### Solution
```python
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sk