  The identification of predictive biomarkers from a large scale of covariates
for subgroup analysis has attracted fundamental attention in medical research.
In this article, we propose a generalized penalized regression method with a
novel penalty function, for enforcing the hierarchy structure between the
prognostic and predictive effects, such that a nonzero predictive effect must
induce its ancestor prognostic effects being nonzero in the model. Our method
is able to select useful predictive biomarkers by yielding a sparse,
interpretable, and predictable model for subgroup analysis, and can deal with
different types of response variable such as continuous, categorical, and
time-to-event data. We show that our method is asymptotically consistent under
some regularized conditions. To minimize the generalized penalized regression
model, we propose a novel integrative optimization algorithm by integrating the
majorization-minimization and the alternating direction method of multipliers,
which is named after \texttt{smog}. The enriched simulation study and real case
study demonstrate that our method is very powerful for discovering the true
predictive biomarkers and identifying subgroups of patients.

Methodology;Machine Learning