Answers.Body: <p>I don't find it inherently suspect to report results in the form of summary statistics like the MCC, F1 score, or chi-squared statistic or p-value. Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data or run all possible statistical tests. You mention that you'd have been happier seeing precision and recall, but those themselves are just other choices of summary statistics from a wide range of options. Different readers want different analyses or levels of detail. I'd say it's typically good research practice to release as much underlying data as possible when publishing, but it's quite common not to, so failure to include raw confusion matrices doesn't by itself strike me as a red flag.</p>
  <p>Now, it's always possible that the authors have purposefully omitted particular representations of their results in order to make their method look more impressive than it is. There is of course no justification for bad-faith selective presentation of the statistics, but all authors must be selective in some way when communicating to their readers.</p>
  <p>In this particular case, I agree it might have been interesting and informative to see if there is any bias toward Type 1 or Type 2 errors. The authors found the summary statistics sufficient, however - from their domain perspective, they're not particularly interested in whether they're incorrectly predicting that survivors will die or that the soon-to-die will survive. The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application, which would be fraught with ethical concerns.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T18:19:43.377Z
Answers.Text: "Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data"  They can include huge diagrams but not 4 values because of space?  Really?
Answers.UserDisplayName: JimmyJames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T18:37:40.220Z
Answers.Text: "The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application,." That's sort of the problem I'm pointing to.  The paper is basically saying, "this is doable because we did it."  But did they?  They claim their results performed 11% better than an existing benchmark, which is *what*, exactly? I'm also not completely sure whether they used 2020 data in their evaluation which would definitely throw the general applicability of this into question.
Answers.UserDisplayName: JimmyJames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T18:43:37.920Z
Answers.Text: @JimmyJames Sometimes more data detracts from the message - the authors don't seem to care at all about Type 1 vs. Type 2 errors, so they're not going to point the reader's focus in that direction. The paper reports performance statistics from several *dozen* models, reporting all the underlying confusion matrices would almost surely have been overkill. I don't really see how their choice of benchmark or practical use case relates to your original question, I'm just pointing out that omitting data may (or may not) be done for practical or well-justified reasons.
Answers.UserDisplayName: Nuclear Hoagie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T19:07:18.017Z
Answers.Text: "several dozen models" I guess I missed that.  How do they only end up with one MCC value, then?
Answers.UserDisplayName: JimmyJames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T19:10:13.447Z
Answers.Text: @JimmyJames They get one final model, but do report a lot more performance stats in the model tuning section. You could argue all that data should be reported too, but you have to draw the line somewhere.
Answers.UserDisplayName: Nuclear Hoagie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T19:17:38.600Z
Answers.Text: I get they can include everything but if they provide the confusion matrix, the MCC can be calculated, as well as F1 and other summary stats.  We cannot work back to the confusion matrix from MCC, however.  It seems a little like buying groceries and getting a receipt that just says 'food' and a total at the bottom.
Answers.UserDisplayName: JimmyJames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T19:23:50.200Z
Answers.Text: @JimmyJames True, but summaries stats do have their use as summaries - only if I am very invested in a paper would I calculate further statistics from raw confusion matrices. It ultimately depends on what you're trying to do - if I just want to know if I spent more on food than last month, I don't need the itemized receipt, just the total. For good or bad, the authors present their method as a high-level summary. It hides some information, but streamlines what remains. You can always write the authors and ask for more detail.
Answers.UserDisplayName: Nuclear Hoagie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T19:27:36.947Z
Answers.Text: " if I just want to know if I spent more on food than last month, I don't need the itemized receipt, just the total" But if you want to know why you spent more, or if the charge was accurate, you are SOL. You can't question things if you aren't told about them.  Of course, this isn't even a repeatable study so maybe I need to stop treating it like hard science.
Answers.UserDisplayName: JimmyJames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T17:13:30.717Z
Answers.LastActivityDate: 2024-01-02T17:13:30.717Z
Answers.UserDisplayName: Nuclear Hoagie
Body: <p>Over the last few days I have been seeing a lot of buzz and news articles about an AI death predictor that is 'highly accurate' based on a 'life story'.  Due accuracy being such a <a href="https://developers.google.com/machine-learning/crash-course/classification/accuracy" rel="nofollow noreferrer">poor indicator of predictive power</a>, I tend to be highly suspicious of claims that focus on it.  For example, I can write function <code>return True</code> which will be extremely accurate for many questions such as, &quot;is person X on earth?&quot; I realize that most mainstream media articles are not going to get into more meaningful statistics so that doesn't mean it's necessarily bogus so I decided to look into the specifics of the claim more closely.  I was able to access the article <a href="https://www.nature.com/articles/s43588-023-00573-5" rel="nofollow noreferrer">here</a> from my home computer (but not through my work VPN.)</p>
  <p>I was a little underwhelmed by what I found.  First off, I cannot find a <a href="https://developers.google.com/machine-learning/glossary#confusion-matrix" rel="nofollow noreferrer">confusion matrix</a> or values that could be used to build one.  Secondly, I cannot find any reference to the <a href="https://nypost.com/2023/12/20/tech/ai-death-calculator-predicts-when-youll-die-its-extremely-accurate/" rel="nofollow noreferrer">&quot;78% correctness&quot;</a> mentioned in some of the articles I have seen.  I also don't see anything like the claims that this will predict &quot;when you'll die&quot; given your life story.  It seems clear that many of the news articles misrepresent the actual study, but the paper itself has some puzzling aspects.</p>
  <p>The actual prediction was whether a person would survive for the 4 years after 2016 based on a number of factors.  However, I can't find the values for: true positives (TP), true negatives (TN), false positives (FP), or false negatives (FN).  All I can find is the mean-corrected <a href="https://en.wikipedia.org/wiki/Phi_coefficient" rel="nofollow noreferrer">Matthew's correlation coefficient</a> (C-MCC) which is given as 0.41 which ranges from -1 to 1 where a value of 1 is completely correct, 0 is random chance, and values below 0 are worse than random.</p>
  <p>Assuming I haven't just missed something (I searched visually and on key terms,) this seems a little strange.  The MCC is calculated from these values but you cannot compute those inputs if all you have is the MCC value.  The closest it seems to get is a (pretty confusing, IMO) plot which seems to show a lot of incorrect predictions and kind of looks to me like it mostly predicts that older people are more likely to die.</p>
  <p>Question: Is it suspect to exclude the values that were used to calculate the MCC?  This seems pretty basic to me.  At the very least I would expect a paper of this type to provide precision and recall.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T13:11:01.600Z
Comments.Text: Too many questions in the same post. It seems that your main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc. If your questions are just subquestions of a main question, can you please put your main specific question in the title? Thanks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T16:05:29.537Z
Comments.Text: "main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc" I understand how those relate.  Anyone can look that up.  I'm trying to understand why the core statistics seem to be obfuscated in the paper.
Comments.UserDisplayName: JimmyJames
Tags: <research><prediction>
Title: Is there any justification for excluding core statistics in 'AI Death Predictor' Paper?
UserDisplayName: JimmyJames
ThreadText: # Is there any justification for excluding core statistics in 'AI Death Predictor' Paper?
  
  (Asked by: username_0 on 2023-12-22)
  
  Over the last few days I have been seeing a lot of buzz and news articles about an AI death predictor that is 'highly accurate' based on a 'life story'. Due accuracy being such a [poor indicator of predictive power](https://developers.google.com/machine-learning/crash-course/classification/accuracy), I tend to be highly suspicious of claims that focus on it. For example, I can write function `return True` which will be extremely accurate for many questions such as, "is person X on earth?" I realize that most mainstream media articles are not going to get into more meaningful statistics so that doesn't mean it's necessarily bogus so I decided to look into the specifics of the claim more closely. I was able to access the article [here](https://www.nature.com/articles/s43588-023-00573-5) from my home computer (but not through my work VPN.)
  
  I was a little underwhelmed by what I found. First off, I cannot find a [confusion matrix](https://developers.google.com/machine-learning/glossary#confusion-matrix) or values that could be used to build one. Secondly, I cannot find any reference to the ["78% correctness"](https://nypost.com/2023/12/20/tech/ai-death-calculator-predicts-when-youll-die-its-extremely-accurate/) mentioned in some of the articles I have seen. I also don't see anything like the claims that this will predict "when you'll die" given your life story. It seems clear that many of the news articles misrepresent the actual study, but the paper itself has some puzzling aspects.
  
  The actual prediction was whether a person would survive for the 4 years after 2016 based on a number of factors. However, I can't find the values for: true positives (TP), true negatives (TN), false positives (FP), or false negatives (FN). All I can find is the mean-corrected [Matthew's correlation coefficient](https://en.wikipedia.org/wiki/Phi_coefficient) (C-MCC) which is given as 0.41 which ranges from -1 to 1 where a value of 1 is completely correct, 0 is random chance, and values below 0 are worse than random.
  
  Assuming I haven't just missed something (I searched visually and on key terms,) this seems a little strange. The MCC is calculated from these values but you cannot compute those inputs if all you have is the MCC value. The closest it seems to get is a (pretty confusing, IMO) plot which seems to show a lot of incorrect predictions and kind of looks to me like it mostly predicts that older people are more likely to die.
  
  Question: Is it suspect to exclude the values that were used to calculate the MCC? This seems pretty basic to me. At the very least I would expect a paper of this type to provide precision and recall.
  
  **Tags:** research, prediction
  > **Comments:**
  > **username_1**: Too many questions in the same post. It seems that your main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc. If your questions are just subquestions of a main question, can you please put your main specific question in the title? Thanks.  
  > **username_0**: "main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc" I understand how those relate.  Anyone can look that up.  I'm trying to understand why the core statistics seem to be obfuscated in the paper.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-01-02. Score: 1
  
  I don't find it inherently suspect to report results in the form of summary statistics like the MCC, F1 score, or chi-squared statistic or p-value. Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data or run all possible statistical tests. You mention that you'd have been happier seeing precision and recall, but those themselves are just other choices of summary statistics from a wide range of options. Different readers want different analyses or levels of detail. I'd say it's typically good research practice to release as much underlying data as possible when publishing, but it's quite common not to, so failure to include raw confusion matrices doesn't by itself strike me as a red flag.
  
  Now, it's always possible that the authors have purposefully omitted particular representations of their results in order to make their method look more impressive than it is. There is of course no justification for bad-faith selective presentation of the statistics, but all authors must be selective in some way when communicating to their readers.
  
  In this particular case, I agree it might have been interesting and informative to see if there is any bias toward Type 1 or Type 2 errors. The authors found the summary statistics sufficient, however - from their domain perspective, they're not particularly interested in whether they're incorrectly predicting that survivors will die or that the soon-to-die will survive. The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application, which would be fraught with ethical concerns.
  
  > **Comments:**
  > **username_0**: "Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data"  They can include huge diagrams but not 4 values because of space?  Really?  
  > **username_0**: "The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application,." That's sort of the problem I'm pointing to.  The paper is basically saying, "this is doable because we did it."  But did they?  They claim their results performed 11% better than an existing benchmark, which is *what*, exactly? I'm also not completely sure whether they used 2020 data in their evaluation which would definitely throw the general applicability of this into question.  
  > **username_2**: @JimmyJames Sometimes more data detracts from the message - the authors don't seem to care at all about Type 1 vs. Type 2 errors, so they're not going to point the reader's focus in that direction. The paper reports performance statistics from several *dozen* models, reporting all the underlying confusion matrices would almost surely have been overkill. I don't really see how their choice of benchmark or practical use case relates to your original question, I'm just pointing out that omitting data may (or may not) be done for practical or well-justified reasons.  
  > **username_0**: "several dozen models" I guess I missed that.  How do they only end up with one MCC value, then?  
  > **username_2**: @JimmyJames They get one final model, but do report a lot more performance stats in the model tuning section. You could argue all that data should be reported too, but you have to draw the line somewhere.  
  > **username_0**: I get they can include everything but if they provide the confusion matrix, the MCC can be calculated, as well as F1 and other summary stats.  We cannot work back to the confusion matrix from MCC, however.  It seems a little like buying groceries and getting a receipt that just says 'food' and a total at the bottom.  
  > **username_2**: @JimmyJames True, but summaries stats do have their use as summaries - only if I am very invested in a paper would I calculate further statistics from raw confusion matrices. It ultimately depends on what you're trying to do - if I just want to know if I spent more on food than last month, I don't need the itemized receipt, just the total. For good or bad, the authors present their method as a high-level summary. It hides some information, but streamlines what remains. You can always write the authors and ask for more detail.  
  > **username_0**: " if I just want to know if I spent more on food than last month, I don't need the itemized receipt, just the total" But if you want to know why you spent more, or if the charge was accurate, you are SOL. You can't question things if you aren't told about them.  Of course, this isn't even a repeatable study so maybe I need to stop treating it like hard science.  
  
  ---
  
  