subscription prediction model- how to define users that didn't have an active subscriber when I have created the training dataset for the subscription classifier, a user with an active subscription was labeled with 1.
I have a dilemma for users with label zero. At first, all the users with no active subscription were labeled with 0. However, a user with no active subscription can be a subscriber in the future and should be predicted by the model. A user can't be in the training set (or val set or test set) with label zero, and after that, to be treated as a potential new subscriber and to be predicted by the model.
In the bottom line:
when to treat a user as a user with label 0, and place him in the training set, and when not to train the model on that user and let him to predicted by the classifer?

Let me make sure I understand your setup. You want to predict likelihood to subscribe. You have data on users, including a dummy for their current subscription status. This dummy is your outcome variable, is that right? Are you by chance using it also as a predictor?

However, a user with no active subscription can be a subscriber in the future and should be predicted by the model.

Should this be the model's job? It sounds like you want to identify a sample of users who can be targeted for subscription. If so, you may want to split the sample of current non-subscribers into two:

*

*A: Users who are not going to subscribe, for whatever reason

*B: Users who are targets for subscription

This has to be done with some combination of product knowledge, clustering techniques, etc. Group A can go into both training and validating. You can hold group B out of training and validation to calculate probabilities of subscription for it.
