Answers.Body: <p>Example 2 and 3 seem symmetric because they are forced to by the model. The Cox proportional hazard relies on the proportional hazards assumption. This basically means that the ratio of the hazards are constant over time. If you were to plot the hazard ratio, it would look like a flat line over all <code>Event_at</code>. Another way of saying this is that the hazard ratios are constant over time.</p>
  <p>How that shows up in a survival plot is a little different because of the relationship between hazards and survival. The survival curves end up following the pattern exemplified in Example 2.</p>
  <p>As for example 3, I would guess that there are few individuals with event times past 3500. This is why you have those large jumps.</p>
  <p><strong>Proportional Hazards</strong></p>
  <p>So what does the proportional hazards assumption mean? Below are some plots of hazard functions to help your intuition on what the proportional hazards assumption means. In all of the following scenarios, the hazard ratio is 2 for blue-to-red. However, the proportional hazards assumption is only true in A, B, and C.</p>
  <p><a href="https://i.sstatic.net/UxBe6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UxBe6.png" alt="Plot of hazards" /></a></p>
  <p>Since the <em>proportion</em> is remaining constant, it can look a little hard to see. To make it easier on the eyes, you can log-transform the hazard functions. Below is a plot that does that.</p>
  <p><a href="https://i.sstatic.net/eB7yG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eB7yG.png" alt="enter image description here" /></a></p>
  <p>As you can see, the hazards are a constant distance from each other for A, B, and C. Basically, the proportional hazards assumption is saying that your scenario looks something like one of those plots.</p>
  <p>In scenario D, the hazard ratio varies over time. At the beginning it is basically HR=1. At the end it is basically HR=4. With the proportional hazards assumption, Cox regression will essentially take the 'average' of the hazards over time. In this case it would give you an estimate of HR=2. Depending on the scenario, this 'smoothing' may be more or less acceptable. But this is basically what the proportional hazards assumption does in the background.</p>
  <p>The proportional hazards assumption is actually more problematic in the cases where the two hazard functions cross each other. Below is an example of that.</p>
  <p><a href="https://i.sstatic.net/CMWZO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CMWZO.png" alt="enter image description here" /></a></p>
  <p>In this scenario, blue has an increased hazard at the start (relative to red) then a decreased at the end. A Cox proportional hazards model would give you HR=1 in this case, because the average over all time points is no relationship.</p>
  <p>So how do you tell if you violate the proportional hazards assumption? You may want to use a stratified Kaplan-Meier by your variables to get an idea of what the hazards look like without stipulating the proportional hazards assumption. You can use the Kaplan-Meier estimator to get a non-parametric estimate of the discrete hazard (i.e. it does not make the proportional hazards assumption). Then you can log-transform those hazards to get a plot like the second. There are also statistical tests and other diagnostics (Schoenfeld residuals) to check this assumption.</p>
  <p>The next question is do those violations matter? This somewhat depends on the question you are trying to answer and how substantial the violations are. In cases where the two log-hazard curves don't cross (like in D), it may be acceptable to smooth over the variations. However, it settings where the curves cross in the final figure, it is probably best to not ignore this problem.</p>
  <p>The original write-up of this answer was <a href="https://github.com/CamDavidsonPilon/lifelines/issues/1168" rel="nofollow noreferrer">here</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-19T13:03:05.787Z
Answers.Text: Please upvote this @pzvich this is a truely excellent response and helps me understand survival analysis
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-19T12:46:39.400Z
Answers.LastActivityDate: 2020-11-19T12:46:39.400Z
Answers.UserDisplayName: pzivich
Body: <p>Hi and sorry for the long post in advance,</p>
  <p>I'm doing a survival analysis of lung cancer patients using Python's lifelines package.
  According to the documentation, the function plot_partial_effects_on_outcome() plots the effect of a covariate on the observer's survival. I do understand that the CoxPH model assumes that the log-hazard of an individual is modelled by a  linear function of their covariates, however, in some cases the effect of these covariates seems to perfect when plotted. I'll illustrate these with three images below describing the issues I have in my data analysis.</p>
  <p>Example 1:
  This one I'm providing as it looks realistic to me, there is a clear difference in the effect of the covariate when positive or negative.
  <img src="https://user-images.githubusercontent.com/25772555/99532844-da5b8500-299c-11eb-9b02-b0a81bfa6887.png" alt="image" /></p>
  <p>Example 2:
  This one just looks gimmicky, my concern is firstly there seems to be a perfect relation between the pathologic stage of cancer and the survival, it's logical that higher stages should impact the survival negatively, however, I don't understand why the graph is so symmetric as the survivability of cancer shouldn't be proportional to the patient cancer staging.
  <img src="https://user-images.githubusercontent.com/25772555/99533165-5ce44480-299d-11eb-90fd-02255a88b9a6.png" alt="image" /></p>
  <p>Example 3:
  Same problems as for example 2, this time with a binary not categorical variable (so I'm less reluctant on disbelieving it), however, the curves do seem to symmetric, almost like it's the same curve but with a standard deviation. I also highlighted a region of interest which I don't understand why both curves stabilize (maybe lack of patients in those regions?)
  <img src="https://user-images.githubusercontent.com/25772555/99533751-4ee2f380-299e-11eb-9770-513204b9e11c.png" alt="image" /></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T15:47:42.140Z
Comments.Text: I would post to cross-validation unless @StupidWolf has done survival analysis. I've never done it, so can't help. Those are very different probabilities. and the drop is simply the tail of the distribution and sampling at event 5000. I wouldn't worry about that- the difference between the top and bottom graph is ... waoooo
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T16:22:24.887Z
Comments.Text: Thank you for your contribution! I'll wait a bit before moving it to cross-validation, just thought that considering the domain it would be more fit here.
Comments.UserDisplayName: beerzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-19T13:03:24.670Z
Comments.Text: You were right to do so. I retract my original note.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-19T13:10:58.057Z
Comments.Text: How come "condition 1" (red line) in the first plot does not start from 1.0? Can it just be the result of this curve being to steep?
Comments.UserDisplayName: haci
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-19T13:32:02.153Z
Comments.Text: That's actually a good observation, and I'm not sure why it's happening. I've looked into it and zoomed the graph, and it's actually not starting at 1, but at 0.9. 
  It may be something to do with the variable not being fit for this type of analysis as there is only one positive case in 496. The only positive case is a patient who died on day 24, so I can see the curve being steep however I have no idea why it doesn't start at 1
Comments.UserDisplayName: beerzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-19T13:48:03.360Z
Comments.Text: The actual hazard ratio of the covariate ´paper_Selected.Mutation.Summary.TP53 p.R280I´ is 147.00275 which is actually the highest for the LUSC analysis. Now my only assumption why this might be correct while it seems strange is that there are actually some patients dying after 0 days, although they do not have this covariate positive.
  Then again, it might just mean that with only 1 positive case in 496 samples this covariate can act more like an outlier than anything. If anyone has a supposition why it doesn't start at 1 I can provide more info.
Comments.UserDisplayName: beerzy
Tags: <python><visualization><cancer>
Title: Survival analysis using CoxPH - Effect of covariates
ThreadText: # Survival analysis using CoxPH - Effect of covariates
  
  (Asked by: username_0 on 2020-11-18)
  
  Hi and sorry for the long post in advance,
  
  I'm doing a survival analysis of lung cancer patients using Python's lifelines package.
  According to the documentation, the function plot\_partial\_effects\_on\_outcome() plots the effect of a covariate on the observer's survival. I do understand that the CoxPH model assumes that the log-hazard of an individual is modelled by a linear function of their covariates, however, in some cases the effect of these covariates seems to perfect when plotted. I'll illustrate these with three images below describing the issues I have in my data analysis.
  
  Example 1:
  This one I'm providing as it looks realistic to me, there is a clear difference in the effect of the covariate when positive or negative.
  ![image](https://user-images.githubusercontent.com/25772555/99532844-da5b8500-299c-11eb-9b02-b0a81bfa6887.png)
  
  Example 2:
  This one just looks gimmicky, my concern is firstly there seems to be a perfect relation between the pathologic stage of cancer and the survival, it's logical that higher stages should impact the survival negatively, however, I don't understand why the graph is so symmetric as the survivability of cancer shouldn't be proportional to the patient cancer staging.
  ![image](https://user-images.githubusercontent.com/25772555/99533165-5ce44480-299d-11eb-90fd-02255a88b9a6.png)
  
  Example 3:
  Same problems as for example 2, this time with a binary not categorical variable (so I'm less reluctant on disbelieving it), however, the curves do seem to symmetric, almost like it's the same curve but with a standard deviation. I also highlighted a region of interest which I don't understand why both curves stabilize (maybe lack of patients in those regions?)
  ![image](https://user-images.githubusercontent.com/25772555/99533751-4ee2f380-299e-11eb-9770-513204b9e11c.png)
  
  **Tags:** python, visualization, cancer
  
  > **Comments:**
  > **username_1**: I would post to cross-validation unless @StupidWolf has done survival analysis. I've never done it, so can't help. Those are very different probabilities. and the drop is simply the tail of the distribution and sampling at event 5000. I wouldn't worry about that- the difference between the top and bottom graph is ... waoooo  
  > **username_0**: Thank you for your contribution! I'll wait a bit before moving it to cross-validation, just thought that considering the domain it would be more fit here.  
  > **username_1**: You were right to do so. I retract my original note.  
  > **username_2**: How come "condition 1" (red line) in the first plot does not start from 1.0? Can it just be the result of this curve being to steep?  
  > **username_0**: That's actually a good observation, and I'm not sure why it's happening. I've looked into it and zoomed the graph, and it's actually not starting at 1, but at 0.9.  It may be something to do with the variable not being fit for this type of analysis as there is only one positive case in 496. The only positive case is a patient who died on day 24, so I can see the curve being steep however I have no idea why it doesn't start at 1  
  > **username_0**: The actual hazard ratio of the covariate ´paper_Selected.Mutation.Summary.TP53 p.R280I´ is 147.00275 which is actually the highest for the LUSC analysis. Now my only assumption why this might be correct while it seems strange is that there are actually some patients dying after 0 days, although they do not have this covariate positive. Then again, it might just mean that with only 1 positive case in 496 samples this covariate can act more like an outlier than anything. If anyone has a supposition why it doesn't start at 1 I can provide more info.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-11-19. Score: 3 | Accepted answer
  
  Example 2 and 3 seem symmetric because they are forced to by the model. The Cox proportional hazard relies on the proportional hazards assumption. This basically means that the ratio of the hazards are constant over time. If you were to plot the hazard ratio, it would look like a flat line over all `Event_at`. Another way of saying this is that the hazard ratios are constant over time.
  
  How that shows up in a survival plot is a little different because of the relationship between hazards and survival. The survival curves end up following the pattern exemplified in Example 2.
  
  As for example 3, I would guess that there are few individuals with event times past 3500. This is why you have those large jumps.
  
  **Proportional Hazards**
  
  So what does the proportional hazards assumption mean? Below are some plots of hazard functions to help your intuition on what the proportional hazards assumption means. In all of the following scenarios, the hazard ratio is 2 for blue-to-red. However, the proportional hazards assumption is only true in A, B, and C.
  
  [![Plot of hazards](https://i.sstatic.net/UxBe6.png)](https://i.sstatic.net/UxBe6.png)
  
  Since the *proportion* is remaining constant, it can look a little hard to see. To make it easier on the eyes, you can log-transform the hazard functions. Below is a plot that does that.
  
  [![enter image description here](https://i.sstatic.net/eB7yG.png)](https://i.sstatic.net/eB7yG.png)
  
  As you can see, the hazards are a constant distance from each other for A, B, and C. Basically, the proportional hazards assumption is saying that your scenario looks something like one of those plots.
  
  In scenario D, the hazard ratio varies over time. At the beginning it is basically HR=1. At the end it is basically HR=4. With the proportional hazards assumption, Cox regression will essentially take the 'average' of the hazards over time. In this case it would give you an estimate of HR=2. Depending on the scenario, this 'smoothing' may be more or less acceptable. But this is basically what the proportional hazards assumption does in the background.
  
  The proportional hazards assumption is actually more problematic in the cases where the two hazard functions cross each other. Below is an example of that.
  
  [![enter image description here](https://i.sstatic.net/CMWZO.png)](https://i.sstatic.net/CMWZO.png)
  
  In this scenario, blue has an increased hazard at the start (relative to red) then a decreased at the end. A Cox proportional hazards model would give you HR=1 in this case, because the average over all time points is no relationship.
  
  So how do you tell if you violate the proportional hazards assumption? You may want to use a stratified Kaplan-Meier by your variables to get an idea of what the hazards look like without stipulating the proportional hazards assumption. You can use the Kaplan-Meier estimator to get a non-parametric estimate of the discrete hazard (i.e. it does not make the proportional hazards assumption). Then you can log-transform those hazards to get a plot like the second. There are also statistical tests and other diagnostics (Schoenfeld residuals) to check this assumption.
  
  The next question is do those violations matter? This somewhat depends on the question you are trying to answer and how substantial the violations are. In cases where the two log-hazard curves don't cross (like in D), it may be acceptable to smooth over the variations. However, it settings where the curves cross in the final figure, it is probably best to not ignore this problem.
  
  The original write-up of this answer was [here](https://github.com/CamDavidsonPilon/lifelines/issues/1168)
  
  > **Comments:**
  > **username_1**: Please upvote this @pzvich this is a truely excellent response and helps me understand survival analysis  
  
  ---
  
  