🕵️ Looking for DPO experts!
text
🕵️ Looking for DPO experts!
new_line
new_line
I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of 
text
I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of 
/*****/
`/*****/`
inline_code
. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
text
. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
new_line
new_line
Any thoughts? Any suggestions to get around this? All discussions are welcome!
text
Any thoughts? Any suggestions to get around this? All discussions are welcome!
🕵️ Looking for DPO experts!

I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of `/*****/`. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?

Any thoughts? Any suggestions to get around this? All discussions are welcome!
🤗
mrfakename davanstrien clem osanseviero lewtun Dlbk lilsyoss gsarti julien-c clefourrier lunarflu larryvrh
5e1e17b6fcf41d740b6996a8
https://cdn-avatars.huggingface.co/v1/production/uploads/1594192845975-5e1e17b6fcf41d740b6996a8.jpeg
Bram Vanroy
BramVanroy
user
6362ef7a19cf373a5fc52d3f
/avatars/4dc63d2be80ce2d1610d2742f42973a1.svg
Robin Smits
robinsmits
user
5f04e8865d08220171a0ad3f
https://cdn-avatars.huggingface.co/v1/production/uploads/1647276617786-5f04e8865d08220171a0ad3f.png
François Remy
FremyCompany
user
6495d5a915d8ef6f01bc75eb
/avatars/245ec3b183919c079f8c5023b3f7ca9f.svg
CultriX
CultriX
user
5f0c746619cb630495b814fd
https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
Lewis Tunstall
lewtun
user
64708d017fd7ecdbd0ea8b5b
/avatars/ce2c27ba75e4a8004e8616a78d5ff44d.svg
LE WANG
wlhgtc
user
60420dccc15e823a685f2b03
https://cdn-avatars.huggingface.co/v1/production/uploads/60420dccc15e823a685f2b03/AOApMWt_jvm9e6XQ2vlrJ.jpeg
Daniel Vila
dvilasuero
user
61b8e2ba285851687028d395
https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/Rq3xWG7mJ3aCRoBsq340h.jpeg
Maxime Labonne
mlabonne
user
618b9a79ba796dc2bf3f4412
https://cdn-avatars.huggingface.co/v1/production/uploads/618b9a79ba796dc2bf3f4412/66ustwEp1EDU_rxcXJBIO.jpeg
Bram van Es
UMCU
user