Answers.Body: <p>There is no overtraining here, it isn't deep learning and there appears no parameterisation. If parameterisation was occurring on the training set that can and does cause over-tightening.</p>
  <p>Essentially, you haven't included a data split for parameterisation, i.e. there is no parameterisation. The theory of a parameterisation is it must be distinct from training (and obviously) testing. It allows parameter tuned without overtightening. Lasso regression will require parameterisation.</p>
  <p>I don't know about the survival regressions for machine learning, but lasso/ridge regression are standard approaches.</p>
  <hr />
  <p>So just to explain, what you do is have a split for parameters. Thus instead of 80:20 split it could be 70:20:10 split where the 10% is used for parameter optimisation. Bad code risks leakage from the parameter split and that can result in over-tightening.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-26T15:17:07.250Z
Answers.LastActivityDate: 2022-07-26T20:01:37.923Z
Answers.LastEditDate: 2022-07-26T20:01:37.923Z
Answers.UserDisplayName: M__
Body: <ol>
  <li>I got the data of stomach cancer from TCGA, I found some different
  expression genes between cancer and normal sample.</li>
  <li>I put aside my normal sample, and divide my cancer sample into
  train(80%) and test(20%) groups.</li>
  <li>I did Unicox (with survival package), lasso (with glmnet package)
  and multicox ( with survival package) on train group, to find out
  which genes are more related to survival.</li>
  <li>I use coefficients of each genes (that I got from multicox), to
  built my model and calculate the risk score. Like that: (expression
  of gene1×coefficient gene1 + expression of gene2×coefficient gene2
  +...).</li>
  <li>I divided my sample in to high risk and low risk groups, to compare
  survival between them, using Kaplan-Meier.</li>
  </ol>
  <p>In training group, Kaplan-Meier shows significant difference between high and low risk group. However when I use my model on my test group, it is not significant. Why this problem occurs? I assumed my model is overfit of my train group, but how can I fix it? Why does this happen even when I use lasso?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-25T21:01:44.170Z
Comments.Text: Do you have sufficient power? Are your hazard rates/ratios very different between train and test? It would be possible to have a model that performs exactly as well as it did in the training data, but that shows up with no significant difference simply due to having a smaller sample size in the test data.
Comments.UserDisplayName: Nuclear Hoagie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-25T21:10:34.833Z
Comments.Text: Hi, thank u for your answer, yes at first I thought it is because of small sample size in test group, so I tried diving samples, 70-30, 60-40, and even 50-50. In all of the them pvalue in Kaplan meier was <0.001 in train group, and in non of the it was significant in test group
Comments.UserDisplayName: Farideh Jafari
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-25T21:15:19.453Z
Comments.Text: Hi @FaridehJafari could you kindly format your question to make it clear?
Comments.UserDisplayName: M__
Tags: <r>
Title: My model doesn't work in test dataset
ThreadText: # My model doesn't work in test dataset
  
  (Asked by: username_0 on 2022-07-25)
  
  1. I got the data of stomach cancer from TCGA, I found some different
     expression genes between cancer and normal sample.
  2. I put aside my normal sample, and divide my cancer sample into
     train(80%) and test(20%) groups.
  3. I did Unicox (with survival package), lasso (with glmnet package)
     and multicox ( with survival package) on train group, to find out
     which genes are more related to survival.
  4. I use coefficients of each genes (that I got from multicox), to
     built my model and calculate the risk score. Like that: (expression
     of gene1×coefficient gene1 + expression of gene2×coefficient gene2
     +...).
  5. I divided my sample in to high risk and low risk groups, to compare
     survival between them, using Kaplan-Meier.
  
  In training group, Kaplan-Meier shows significant difference between high and low risk group. However when I use my model on my test group, it is not significant. Why this problem occurs? I assumed my model is overfit of my train group, but how can I fix it? Why does this happen even when I use lasso?
  
  **Tags:** r
  
  > **Comments:**
  > **username_1**: Do you have sufficient power? Are your hazard rates/ratios very different between train and test? It would be possible to have a model that performs exactly as well as it did in the training data, but that shows up with no significant difference simply due to having a smaller sample size in the test data.  
  > **username_0**: Hi, thank u for your answer, yes at first I thought it is because of small sample size in test group, so I tried diving samples, 70-30, 60-40, and even 50-50. In all of the them pvalue in Kaplan meier was <0.001 in train group, and in non of the it was significant in test group  
  > **username_2**: Hi @username_0 could you kindly format your question to make it clear?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-07-26. Score: 0
  
  There is no overtraining here, it isn't deep learning and there appears no parameterisation. If parameterisation was occurring on the training set that can and does cause over-tightening.
  
  Essentially, you haven't included a data split for parameterisation, i.e. there is no parameterisation. The theory of a parameterisation is it must be distinct from training (and obviously) testing. It allows parameter tuned without overtightening. Lasso regression will require parameterisation.
  
  I don't know about the survival regressions for machine learning, but lasso/ridge regression are standard approaches.
  
  ---
  
  So just to explain, what you do is have a split for parameters. Thus instead of 80:20 split it could be 70:20:10 split where the 10% is used for parameter optimisation. Bad code risks leakage from the parameter split and that can result in over-tightening.
  
  
  ---
  
  