Question: I know this is a very classical question which might be answered many times in this forum, however I could not find any clear answer which explains this clearly from scratch.

Firstly, imgine that my dataset called my\_data has 4 variables such as
my\_data = variable1, variable2, variable3, target\_variable

So, let's come to my problem. I'll explain all my steps and ask your help for where I've been stuck:

```
# STEP1 : split my_data into [predictors] and [targets]

predictors = my_data[[
'variable1',
'variable2',
'variable3'
]]

targets = my_data.target_variable

# STEP2 : import the required libraries

from sklearn import cross_validation
from sklearn.ensemble import RandomForestRegressor

#STEP3 : define a simple Random Forest model attirbutes

model = RandomForestClassifier(n_estimators=100)

#STEP4 : Simple K-Fold cross validation. 3 folds.

cv = cross_validation.KFold(len(my_data), n_folds=3,  random_state=30)

# STEP 5

```

At this step, I want to fit my model based on the training dataset, and then
use that model on test dataset and predict test targets. I also want to calculate the required statistics such as MSE, r2 etc. for understanding the performance of my model.

I'd appreciate if someone helps me woth some basic codelines for Step5.

Answer: 
First off, you are using the deprecated package `cross-validation` of scikit library. New package is named `model_selection`. So I am using that in this answer.

Second, you are importing `RandomForestRegressor`, but defining `RandomForestClassifier` in the code. I am taking `RandomForestRegressor` here, because the metrics you want (MSE, R2 etc) are only defined for regression problems, not classification.

There are multiple ways to do what you want. I assume that since you are trying to use the KFold cross-validation here, you want to use the left-out data of each fold as test fold. To accomplish this, we can do:

```
predictors = my_data[[
'variable1',
'variable2',
'variable3'
]]

targets = my_data.target_variable

from sklearn import model_selection
from sklearn.ensemble import RandomForestRegressor
from sklearn import metrics

model = RandomForestRegressor(n_estimators=100)

cv = model_selection.KFold(n_splits=3)

for train_index, test_index in kf.split(predictors):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = predictors[train_index], predictors[test_index]
    y_train, y_test = targets[train_index], targets[test_index]

    # For training, fit() is used
    model.fit(X_train, y_train)

    # Default metric is R2 for regression, which can be accessed by score()
    model.score(X_test, y_test)

    # For other metrics, we need the predictions of the model
    y_pred = model.predict(X_test)

    metrics.mean_squared_error(y_test, y_pred)
    metrics.r2_score(y_test, y_pred)

```

For all this, documentation is your best friend. And scikit-learn documentation are one of the best I have ever seen. Following links may help you know more about them:

* <http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-evaluating-estimator-performance>
* <http://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics>
* <http://scikit-learn.org/stable/user_guide.html>
Also in the for loop it should be:

```
model = RandomForestRegressor(n_estimators=100)

for train_index, test_index in cv.split(X):

```