How to compare the random forest performance using R-Squared? I've trained a random forests for a regression problem. Now, I want to check if the model is not overfitted. I have tuned the parameters and then compared the R-Squared of Train and Test dataset as below
test_predict <- model%>% predict(test_data) %>% predictions
R2_test <- 1 - (sum((test_actual-test_predict )^2)/sum((test_actual-mean(test_actual))^2))

train_predict <- model%>% predict(train_data) %>% predictions
R2_train <- 1 - (sum((train_actual-train_predict )^2)/sum((train_actual-mean(train_actual))^2))

The R-Squared on train dataset is significantly higher (close to 0.9) which made think the model is overfit. Then I came across to this question Random Forest - How to handle overfitting. It is saying that predict(model, newdata=train) "treats your training data as if it was a new dataset, and runs the observations down each tree. This will result in an artificially close correlation between the predictions and the actuals, since the RF algorithm generally doesn't prune the individual trees, relying instead on the ensemble of trees to control overfitting."
I don't know what would be the best way to compare the performance of model between train and test dataset.

The gist of $R^2$ is that you compare your performance to a naïve model that always guesses the same value.
$$R^2 = 1-\dfrac{\sum\big(y_i - \hat y_i \big)^2}{\sum\big(y_i - \bar y\big)^2}$$
In the numerator, we subtract the predictions from our model from the observed value. The closer to the observed values are predictions are, the better.
The the denominator, we do the same, but we consider a model that always guesses the mean of the observed $y$ values. If you have no idea how to model the conditional expected value, the reasonable but naive model would always predict the unconditional expected value.
You know how to calculate the numerator and denominator in your software, so calculate them.
You are likely to be interested in how your model does out-of-sample (OOS), and that follows similar logic.
$$R^2_{OOS} = 1-\dfrac{\sum\big(y_{i, OOS} - \hat y_{i, OOS} \big)^2}{\sum\big(y_{i, OOS} - \bar y_{in}\big)^2}$$.
The trick is to keep the denominator as the square loss of a model that naively guesses the same value every time, and the value you guess is still the in-sample mean, since that's the model learned at training time. Don't use the out-of-sample $\bar y$, since you don't know it. (If you knew the $y_i$-values that you would need to calculate out-of-sample $\bar y$, you wouldn't need a model to predict those values. You'd just look them up and have perfect predictions.)
