Body: <p>I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be just 0 and 1, determining if the word level translation is good or bad on both the source and target sides.</p>
  <p>To begin, I used XLMRoberta, as I thought it would be best suited for my problem. First, I just trained normally using nothing fancy, but the model overfits after just 1-2 epochs, as I have very little data to fine-tune on (approx 7k).</p>
  <p>I decided to freeze the BERT layers and just train the classifier weights, but it performed worse.</p>
  <p>I thought of adding a more dense network on top of BERT, but I am not sure if it would work well or not.</p>
  <p>One more thought that occurred to me was data augmentation, where I increased the size of my data by multiple factors, but that performed badly as well. (Also, I am not sure what should be the proper number to increase the data size with augmented data.)</p>
  <p>Can you please suggest which approach could be suitable here and if I am doing something wrong? Shall I just use all the layers for my data or freezing is actually a good option? Or you suspect I am ruining somewhere in the code and this is not what is expected.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-10T09:10:29.607Z
Comments.Text: What do you mean by "I just trained normally using nothing fancy"? Do you mean you're using supervised learning with a labelled dataset? Or are you using self-supervised learning? Or maybe semi-supervised learning, as you wrote at the beginning of your post? It's not clear how much data you're using and for what. Are you just fine-tuning this "XLMRoberta", or training it from scratch? It seems that you also tried to fine-tune it, so I assume that you also trained it from scratch.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-10T09:10:33.190Z
Comments.Text: In both cases, you need to explain 1. how much **training**, **validation** and **test** data do you have 1.1. when you train from scratch, 1.2. when you fine-tune.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-10T14:43:42.220Z
Comments.Text: Thanks @nbro , sorry for the less information. By nothing fancy, i meant i fine tuned the xlmroberta for the problem without augmentation, just the normal training dataset(7k) and validated for 1k dataset. I havent used semi supervised learning as of now, i am just doing a typical supervised fashion. I cannot train the model from scratch as i have a little dataset , so i was trying ti improve my finetuning with augmented dataset, which did not helped.
Comments.UserDisplayName: Tarun Bhatia
Tags: <neural-networks><machine-learning><deep-learning><natural-language-processing><bert>
Title: Fine tuning BERT for token level classification
UserDisplayName: Tarun Bhatia
ThreadText: # Fine tuning BERT for token level classification
  
  (Asked by: username_0 on 2021-12-07)
  
  I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be just 0 and 1, determining if the word level translation is good or bad on both the source and target sides.
  
  To begin, I used XLMRoberta, as I thought it would be best suited for my problem. First, I just trained normally using nothing fancy, but the model overfits after just 1-2 epochs, as I have very little data to fine-tune on (approx 7k).
  
  I decided to freeze the BERT layers and just train the classifier weights, but it performed worse.
  
  I thought of adding a more dense network on top of BERT, but I am not sure if it would work well or not.
  
  One more thought that occurred to me was data augmentation, where I increased the size of my data by multiple factors, but that performed badly as well. (Also, I am not sure what should be the proper number to increase the data size with augmented data.)
  
  Can you please suggest which approach could be suitable here and if I am doing something wrong? Shall I just use all the layers for my data or freezing is actually a good option? Or you suspect I am ruining somewhere in the code and this is not what is expected.
  
  **Tags:** neural-networks, machine-learning, deep-learning, natural-language-processing, bert
  > **Comments:**
  > **username_1**: What do you mean by "I just trained normally using nothing fancy"? Do you mean you're using supervised learning with a labelled dataset? Or are you using self-supervised learning? Or maybe semi-supervised learning, as you wrote at the beginning of your post? It's not clear how much data you're using and for what. Are you just fine-tuning this "XLMRoberta", or training it from scratch? It seems that you also tried to fine-tune it, so I assume that you also trained it from scratch.  
  > **username_1**: In both cases, you need to explain 1. how much **training**, **validation** and **test** data do you have 1.1. when you train from scratch, 1.2. when you fine-tune.  
  > **username_0**: Thanks @nbro , sorry for the less information. By nothing fancy, i meant i fine tuned the xlmroberta for the problem without augmentation, just the normal training dataset(7k) and validated for 1k dataset. I havent used semi supervised learning as of now, i am just doing a typical supervised fashion. I cannot train the model from scratch as i have a little dataset , so i was trying ti improve my finetuning with augmented dataset, which did not helped.  
  
  ---
  
  ## 0 answers
  
  