Answers.Body: <p>OpenAI decided to discontinue its GPT text detection tool mainly due to its <a href="https://winbuzzer.com/2023/07/25/openai-silently-discontinues-ai-text-detection-tool-due-to-low-accuracy-xcxwbn/" rel="nofollow noreferrer">low accuracy</a>. This high rate of errors suggests that distinguishing between human and LLM-written content remains a significant challenge, particularly because models like GPT are continuously evolving and becoming better at mimicking human writing styles.</p>
  <p>For authorship attribution techniques which actually belongs to traditional NLP you may refer papers and other references in <a href="https://en.wikipedia.org/wiki/Stylometry" rel="nofollow noreferrer">stylometry</a> analyzing linguistic features to determine authorship. It is widely used in literature to attribute anonymous works to known authors.</p>
  <p>For recent LLM based stylometry see Kumarage et al's <a href="https://ar5iv.org/html/2308.07305" rel="nofollow noreferrer"><em>Neural Authorship Attribution: Stylometric Analysis on Large Language Models</em></a> and Uchendu st al's <a href="https://aclanthology.org/2020.emnlp-main.673/" rel="nofollow noreferrer">Authorship Attribution for Neural Text Generation</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-08T17:13:53.537Z
Answers.Text: Thanks, maybe I am wrong about this but every paper on Authorship Attribution in regards to the LLMs does say that it achieves higher than 70% accuracy which is pretty good in my opinion but how is that possible? Even the paper I attached says it has higher than 99% accuracy with its newly developed CheckGPT.
Answers.UserDisplayName: Atrox
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T05:38:42.190Z
Answers.Text: It's important to note that high accuracy figures like the ones you mentioned might be contingent on the specific LLMs used during testing or the very specific training datasets which might not cover all real-world usages . If the detection method is tuned to identify a particular LLM's unique patterns (such as GPT-3 or GPT-4), it could achieve much higher accuracy in that context. However, these methods might not generalize as well to detecting text generated by newer or differently trained models or areas outside of its training, which is a classic overfitting concern. Hope this clarifies.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-07T23:55:23.980Z
Answers.LastActivityDate: 2024-12-07T23:55:23.980Z
Answers.UserDisplayName: cinch
Body: <p>I saw OpenAI discontinue their GPT detection tool even though is was performing fairly well according to this paper:
  <a href="https://doi.org/10.48550/arXiv.2306.05524" rel="nofollow noreferrer">https://doi.org/10.48550/arXiv.2306.05524</a></p>
  <p>Why to discontinue it, is it really impossible to find out if the text is LLM or human written?</p>
  <p>Which methods would you advice to use to distinguish between the LLM written and human written?</p>
  <p>Any papers you would advise reading on authorship attribution?</p>
  
Tags: <deep-learning><natural-language-processing><reference-request><papers><large-language-models>
Title: Which factors do distinguish between human and LLMs written text at the moment?
UserDisplayName: Atrox
ThreadText: # Which factors do distinguish between human and LLMs written text at the moment?
  
  (Asked by: username_0 on 2024-12-07)
  
  I saw OpenAI discontinue their GPT detection tool even though is was performing fairly well according to this paper:
  <https://doi.org/10.48550/arXiv.2306.05524>
  
  Why to discontinue it, is it really impossible to find out if the text is LLM or human written?
  
  Which methods would you advice to use to distinguish between the LLM written and human written?
  
  Any papers you would advise reading on authorship attribution?
  
  **Tags:** deep-learning, natural-language-processing, reference-request, papers, large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-07. Score: 1
  
  OpenAI decided to discontinue its GPT text detection tool mainly due to its [low accuracy](https://winbuzzer.com/2023/07/25/openai-silently-discontinues-ai-text-detection-tool-due-to-low-accuracy-xcxwbn/). This high rate of errors suggests that distinguishing between human and LLM-written content remains a significant challenge, particularly because models like GPT are continuously evolving and becoming better at mimicking human writing styles.
  
  For authorship attribution techniques which actually belongs to traditional NLP you may refer papers and other references in [stylometry](https://en.wikipedia.org/wiki/Stylometry) analyzing linguistic features to determine authorship. It is widely used in literature to attribute anonymous works to known authors.
  
  For recent LLM based stylometry see Kumarage et al's [*Neural Authorship Attribution: Stylometric Analysis on Large Language Models*](https://ar5iv.org/html/2308.07305) and Uchendu st al's [Authorship Attribution for Neural Text Generation](https://aclanthology.org/2020.emnlp-main.673/).
  
  > **Comments:**
  > **username_0**: Thanks, maybe I am wrong about this but every paper on Authorship Attribution in regards to the LLMs does say that it achieves higher than 70% accuracy which is pretty good in my opinion but how is that possible? Even the paper I attached says it has higher than 99% accuracy with its newly developed CheckGPT.  
  > **username_1**: It's important to note that high accuracy figures like the ones you mentioned might be contingent on the specific LLMs used during testing or the very specific training datasets which might not cover all real-world usages . If the detection method is tuned to identify a particular LLM's unique patterns (such as GPT-3 or GPT-4), it could achieve much higher accuracy in that context. However, these methods might not generalize as well to detecting text generated by newer or differently trained models or areas outside of its training, which is a classic overfitting concern. Hope this clarifies.  
  
  ---
  
  