Body: <p>Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the pretraining after RLHF with some additional documents, e.g. high-quality scientific papers (e.g. the whole arXiv), yielding M3.</p>
  <p>My question is: Would there be a difference in the quality of answers to scientific questions (that are covered in the additional documents), between M3 (pretraining+RLHF+pretraining) and a model M4 for which pretraining with the additional documents was continued immediately on M1 (pretraining+pretraining+RLHF)?</p>
  <p>The question concerns the interference of pretraining and RLHF. RLHF of course has to be performed on a pretrained model (and doesn't affect the linguistic and implicit world knowledge of the pretrained model too much), but continuation of pretraining only after RLHF might cause more trouble: both general world and aligned knowledge might be reduced, and scientific knowledge not so much advanced.</p>
  <p>Is there an argument that makes it plausible that pretraining+RLHF+pretraining should work well -- or on the contrary would not work at all?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-16T09:45:54.250Z
Comments.Text: personally, i think that M3 would perform worse than M4, because RLHF it's aligning the "output structure" to be human-like, or in other words, to be more human friendly. However, this structure is far from the scientific-paper language structure, thus the autoregressive nature will have a hard time at the beginning, thus having to go ever further from the initial pretraining (pretty much like a continual learning problem)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-16T11:12:26.490Z
Comments.Text: I think this is hard to assess. As in there is no objective assessment of how models M2 onwards perform. So could you please suggest what performance metric or measurements would be important to you in M3 and M4?
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><transformer><unsupervised-learning><pretrained-models><rlhf>
Title: Can pretraining be continued after RLHF?
UserDisplayName: Hans-Peter Stricker
ThreadText: # Can pretraining be continued after RLHF?
  
  (Asked by: username_0 on 2023-08-16)
  
  Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the pretraining after RLHF with some additional documents, e.g. high-quality scientific papers (e.g. the whole arXiv), yielding M3.
  
  My question is: Would there be a difference in the quality of answers to scientific questions (that are covered in the additional documents), between M3 (pretraining+RLHF+pretraining) and a model M4 for which pretraining with the additional documents was continued immediately on M1 (pretraining+pretraining+RLHF)?
  
  The question concerns the interference of pretraining and RLHF. RLHF of course has to be performed on a pretrained model (and doesn't affect the linguistic and implicit world knowledge of the pretrained model too much), but continuation of pretraining only after RLHF might cause more trouble: both general world and aligned knowledge might be reduced, and scientific knowledge not so much advanced.
  
  Is there an argument that makes it plausible that pretraining+RLHF+pretraining should work well -- or on the contrary would not work at all?
  
  **Tags:** reinforcement-learning, transformer, unsupervised-learning, pretrained-models, rlhf
  
  > **Comments:**
  > **username_1**: personally, i think that M3 would perform worse than M4, because RLHF it's aligning the "output structure" to be human-like, or in other words, to be more human friendly. However, this structure is far from the scientific-paper language structure, thus the autoregressive nature will have a hard time at the beginning, thus having to go ever further from the initial pretraining (pretty much like a continual learning problem)  
  > **username_2**: I think this is hard to assess. As in there is no objective assessment of how models M2 onwards perform. So could you please suggest what performance metric or measurements would be important to you in M3 and M4?  
  
  ---
  
  ## 0 answers
  
  