Choice on pretrained model and fine-tuning.

by Avditvs - opened Jun 5

Jun 5

Hi !
The technical paper does not really elaborate on the choice of the backbone (snowflake model) for training the classifier as well as why it was frozen. Could you give more details about the implementation choices ?

anton-l

HuggingFaceFW org Jun 14

Hi @Avditvs ! We've experimented with RoBERTa, mixedbread-ai/mxbai-embed-large-v1, and the snowflake models. Due to significant amounts of noise (+/- 1 point) in the Llama annotations, a frozen encoder helped prevent overfitting, and (counter-intuitively) a retrieval-focused snowflake model worked best. Also snowflake-arctic-embed-m performed just as well as snowflake-arctic-embed-l, so we went with it to save on compute.

liyucheng

Nov 23

@anton-l thanks for this elaboration! any consideration to use regression instead of classification?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment