content.raw: 🕵️ Looking for DPO experts!
content.type: text
content.value: 🕵️ Looking for DPO experts!
content.type: new_line
content.type: new_line
content.raw: I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of 
content.type: text
content.value: I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of 
content.code: /*****/
content.raw: `/*****/`
content.type: inline_code
content.raw: . So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
content.type: text
content.value: . So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
content.type: new_line
content.type: new_line
content.raw: Any thoughts? Any suggestions to get around this? All discussions are welcome!
content.type: text
content.value: Any thoughts? Any suggestions to get around this? All discussions are welcome!
rawContent: 🕵️ Looking for DPO experts!
  
  I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of `/*****/`. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
  
  Any thoughts? Any suggestions to get around this? All discussions are welcome!
reactions.reaction: 🤗
reactions.users: mrfakename davanstrien clem osanseviero lewtun Dlbk lilsyoss gsarti julien-c clefourrier lunarflu larryvrh
commentators._id: 5e1e17b6fcf41d740b6996a8
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594192845975-5e1e17b6fcf41d740b6996a8.jpeg
commentators.fullname: Bram Vanroy
commentators.name: BramVanroy
commentators.type: user
commentators._id: 6362ef7a19cf373a5fc52d3f
commentators.avatarUrl: /avatars/4dc63d2be80ce2d1610d2742f42973a1.svg
commentators.fullname: Robin Smits
commentators.name: robinsmits
commentators.type: user
commentators._id: 6495d5a915d8ef6f01bc75eb
commentators.avatarUrl: /avatars/245ec3b183919c079f8c5023b3f7ca9f.svg
commentators.fullname: CultriX
commentators.name: CultriX
commentators.type: user
commentators._id: 5f04e8865d08220171a0ad3f
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647276617786-5f04e8865d08220171a0ad3f.png
commentators.fullname: François Remy
commentators.name: FremyCompany
commentators.type: user
commentators._id: 5f0c746619cb630495b814fd
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
commentators.fullname: Lewis Tunstall
commentators.name: lewtun
commentators.type: user
commentators._id: 64708d017fd7ecdbd0ea8b5b
commentators.avatarUrl: /avatars/ce2c27ba75e4a8004e8616a78d5ff44d.svg
commentators.fullname: LE WANG
commentators.name: wlhgtc
commentators.type: user
commentators._id: 60420dccc15e823a685f2b03
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60420dccc15e823a685f2b03/AOApMWt_jvm9e6XQ2vlrJ.jpeg
commentators.fullname: Daniel Vila
commentators.name: dvilasuero
commentators.type: user
commentators._id: 61b8e2ba285851687028d395
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/Rq3xWG7mJ3aCRoBsq340h.jpeg
commentators.fullname: Maxime Labonne
commentators.name: mlabonne
commentators.type: user
commentators._id: 618b9a79ba796dc2bf3f4412
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618b9a79ba796dc2bf3f4412/66ustwEp1EDU_rxcXJBIO.jpeg
commentators.fullname: Bram van Es
commentators.name: UMCU
commentators.type: user