Answers.Body: <p>It is difficult. At my previous company I (NLP Engineer) was replaced by an LLM, and looking for NLP jobs, everything was geared towards experience with LLMs. &quot;Traditional&quot; NLP methods are not really <em>en vogue</em> enough at present, it seems.</p>
  <p>However, my own view is that this is a temporary bubble. There is a lot of hype, and over-promises about the capabilities of LLMs. At some point, I expect this to calm down, and there to be a falling back on tried and tested methods which aren't as &quot;sexy&quot;, but do actually work.</p>
  <p>I do realise this is perhaps not a very popular view, but LLMs are basically just smoke and mirrors, albeit very sophisticated.</p>
  <p>So, for the near future it might be difficult to find a job in the NLP area with more traditional skills, but my prediction would be that this will change in the longer term; and even now there are companies who don't want to use LLMs for various reasons (compliance, data protection, etc). Academia might be a better bet for now than industry, as it seems less susceptible to the hype.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-06T18:42:52.110Z
Answers.Text: I don't believe that we will be falling back to "traditional NLP" (e.g. Bag of words) if that's what you mean, Transformers are here to stay. But perhaps with improvements in efficiency & pre-training will make things more accessible.
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-07T20:04:54.260Z
Answers.Text: Bag of words is not traditional NLP. I was thinking of syntactic parsing etc.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-06T09:43:08.090Z
Answers.LastActivityDate: 2024-12-06T09:43:08.090Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>While LLMs like ChatGPT have indeed reshaped NLP, they have not killed NLP research. Instead, the field has shifted in terms of focus and the right niche opportunities available for researchers, even those outside the biggest organizations.</p>
  <p>LLMs like ChatGPT are powerful but often brittle when faced with out-of-distribution (OOD) inputs or nuanced domain-specific higher-level rule-based traditional symbolic AI and NLP tasks. For instance, many languages and dialects are underrepresented or near-extinct and NLP research for these is critical and often does not require massive resources. Real world NLP applications often require domain-specific fine-tuning. For example, legal, medical, and financial NLP involve challenges that general-purpose models like ChatGPT cannot solve out of the box.</p>
  <p>Multi-Modal NLP to combine text with other modalities (e.g., vision, audio) is another evolving domain where new contributions are welcome. Similarly combining NLP with fields like social sciences, mathematics, cognitive science, or healthcare opens new avenues that are less reliant on current ChatGPT like LLMs.</p>
  <p>Finally addressing imminent and difficult NLP issues related to explainability and interpretability, fairness, bias, societal risks and ethical concerns is also a burgeoning area of study.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-06T18:48:47.150Z
Answers.Text: Thanks but I should've mentioned I'm explicitly interested in transformers.
  Like how are modern transformer AI researchers working around ChatGPT?
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-06T06:37:56.733Z
Answers.LastActivityDate: 2024-12-06T06:37:56.733Z
Answers.UserDisplayName: cinch
Body: <p>I'm considering getting into NLP research (that is AI research into Transformers specifically) but my friend made this point which got me concerned, &quot;he said NLP research is dead b/c of ChatGPT&quot;. We all know ChatGPT is enormous and highly versatile. Is it realistic for NLP researchers to make meaningful contributions without working at: google, amazon, ivy leagues, facebook, OpenAI/Microsoft etc...</p>
  <p>P.S. For example recently saw <a href="https://arxiv.org/pdf/2212.01853" rel="nofollow noreferrer">the Vega2 paper</a> 2nd place on <a href="https://super.gluebenchmark.com/leaderboard" rel="nofollow noreferrer">SuperGlue leader board</a> used 320 A100s for &gt; 30 days. Looking at rental prices online that should cost about $300,000!</p>
  
Tags: <natural-language-processing><academia>
Title: How do modern Transformer AI researchers compete with LLMs like ChatGPT?
UserDisplayName: profPlum
ThreadText: # How do modern Transformer AI researchers compete with LLMs like ChatGPT?
  
  (Asked by: username_0 on 2024-12-06)
  
  I'm considering getting into NLP research (that is AI research into Transformers specifically) but my friend made this point which got me concerned, "he said NLP research is dead b/c of ChatGPT". We all know ChatGPT is enormous and highly versatile. Is it realistic for NLP researchers to make meaningful contributions without working at: google, amazon, ivy leagues, facebook, OpenAI/Microsoft etc...
  
  P.S. For example recently saw [the Vega2 paper](https://arxiv.org/pdf/2212.01853) 2nd place on [SuperGlue leader board](https://super.gluebenchmark.com/leaderboard) used 320 A100s for > 30 days. Looking at rental prices online that should cost about $300,000!
  
  **Tags:** natural-language-processing, academia
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-12-06. Score: 1
  
  It is difficult. At my previous company I (NLP Engineer) was replaced by an LLM, and looking for NLP jobs, everything was geared towards experience with LLMs. "Traditional" NLP methods are not really *en vogue* enough at present, it seems.
  
  However, my own view is that this is a temporary bubble. There is a lot of hype, and over-promises about the capabilities of LLMs. At some point, I expect this to calm down, and there to be a falling back on tried and tested methods which aren't as "sexy", but do actually work.
  
  I do realise this is perhaps not a very popular view, but LLMs are basically just smoke and mirrors, albeit very sophisticated.
  
  So, for the near future it might be difficult to find a job in the NLP area with more traditional skills, but my prediction would be that this will change in the longer term; and even now there are companies who don't want to use LLMs for various reasons (compliance, data protection, etc). Academia might be a better bet for now than industry, as it seems less susceptible to the hype.
  
  > **Comments:**
  > **username_0**: I don't believe that we will be falling back to "traditional NLP" (e.g. Bag of words) if that's what you mean, Transformers are here to stay. But perhaps with improvements in efficiency & pre-training will make things more accessible.  
  > **username_1**: Bag of words is not traditional NLP. I was thinking of syntactic parsing etc.  
  
  ---
  
  ### Answer by username_2 on 2024-12-06. Score: 1
  
  While LLMs like ChatGPT have indeed reshaped NLP, they have not killed NLP research. Instead, the field has shifted in terms of focus and the right niche opportunities available for researchers, even those outside the biggest organizations.
  
  LLMs like ChatGPT are powerful but often brittle when faced with out-of-distribution (OOD) inputs or nuanced domain-specific higher-level rule-based traditional symbolic AI and NLP tasks. For instance, many languages and dialects are underrepresented or near-extinct and NLP research for these is critical and often does not require massive resources. Real world NLP applications often require domain-specific fine-tuning. For example, legal, medical, and financial NLP involve challenges that general-purpose models like ChatGPT cannot solve out of the box.
  
  Multi-Modal NLP to combine text with other modalities (e.g., vision, audio) is another evolving domain where new contributions are welcome. Similarly combining NLP with fields like social sciences, mathematics, cognitive science, or healthcare opens new avenues that are less reliant on current ChatGPT like LLMs.
  
  Finally addressing imminent and difficult NLP issues related to explainability and interpretability, fairness, bias, societal risks and ethical concerns is also a burgeoning area of study.
  
  > **Comments:**
  > **username_0**: Thanks but I should've mentioned I'm explicitly interested in transformers. Like how are modern transformer AI researchers working around ChatGPT?  
  
  ---
  
  