Conceptually, can a classification model give good results while a similar regression model gives poor results? I built a really good Random Forest classifier model for predicting whether or not someone will lose weight using several biomarkers as predictors. When I try to build a Random Forest regression model for predicting how much weight someone will lose/gain using the same predictors, it's really really bad.
I am using adjusted R2 to asses the regression model and AUROC to asses the classifier model. Is this disconnect conceptually possible or does this mean I must have messed up somewhere?

Yes this is not just possible, but likely. You are answering two different questions with your models.

*

*Whether or not they gain/lose weight - classification

*How much weight they gain/lose - regression

The first question is about the 'sign' of the weight loss (+ or -) while the second is about the 'sign' and 'magnitude'. The discrepancy between how well your models do suggest you can predict the 'sign' but not the 'magnitude'.
One way to think about this: Imagine if people with biomarker 1 lose: 0.1, 1, 10, 100 lbs while those without it gain 0.1, 1, 10, or 100 lbs. You can see that a classifier works great, but the constraint on the weight loss is extremely poor because of the huge variability between subjects.
So yes this is conceptually possible and does not indicate an error in your modeling (though obviously it doesn't rule that out either).
