Target encoding in test data and target leakage I understand target encoding, which is the average of the target value by category using out-of-fold mean within each fold. although you get slightly different means for the same value of a categorical variable among different folds. An example (source):

Let's say we have 20-fold cross validation. we need somehow to
calculate mean value of the feature for #1 fold using information from
#2-#20 folds only.
So, you take #2-#20 folds, create another cross validation set within
it. Calculate means for every leave-one-out fold. You average these means and apply that
vector for your primary #1 validation set. Repeat that for remaining
19 folds.

My question are:

*

*How using out-of-fold mean within each fold prevents target leakage? Could you give a graphical example or math example or it's too complex with math?

*Related with question (1), I have read that regularization prevents the target leakage when using out-of-fold mean within each fold. Which one is it? Or both prevents target leakage?

*How do I implement this method in the test set? Because I won't have the target variable, and I can't replace the same value of a categorical variable in the test set with the target mean of the train set with that value of the categorical variable. Why? Because the train set for that value has slightly different means. So, which mean do I put for each specific value in the test set?


I think there's confusion about what constitutes target leakage here.  There are two places where this can come up:

*

*The more serious kind, and the kind which I think is generally meant by "target leakage": you incorporate some test set information into your model building, and so your test scores are biased.


*Incorporating training target information into your encoding process, which can lead to too much information available to your eventual model, leading to overfitting.  But if you consider the encoding steps as part of the larger modeling pipeline, this isn't really "wrong", it's just another flavor of overfitting.  You can reduce this effect by various kinds of regularization, see e.g. https://arxiv.org/abs/2104.00629.

I don't understand why your source does nested cross-validation; the outer split seems sufficient to enforce that the target values for a row are taken from an averaged on other samples, reducing the overfitting effect described in point 2 above.  And averaging the means across the inner folds seems likely to be very close to the simple average without the inner folds, assuming the distribution of categories is roughly preserved.
There are other ways to accomplish this too, like smoothing with the prior, which is maybe what you meant in your second question?
To your third question, usually you just use the average for the entire training set.  You wouldn't want to use the targets from the test set if they were available, because that would fall into point 1 above.  The entire modeling procedure assumes that test data is iid with the test data, so your last sentence of question 3 is not any different from any other aspect of the modeling process: you use the training data to make predictions about the test data.
