Roberta Large doesn't train for sentiment classification
The following is my classifier class. I trained it with appropriate code and same tokenizer(roberta-base). It trained well on SST2 2 class sentiment problem. But when i replaced base model with roberta-large (also the tokenizer and dim of linear layer)my classifier is stuck at 50% accuracy and not matter how many epoch i run it doesn't improve. Is the weight file faulty or am i doing something wrong? 



```
class Model(nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.Bert = transformers.RobertaModel.from_pretrained('roberta-base')
        self.fc0 = nn.Linear(768,512)
        self.d0 = nn.Dropout(0.5)
        self.d1 = nn.Dropout(0.5)
        self.fc1 = nn.Linear(512,1)
        nn.init.normal_(self.fc0.weight,std= 0.1)
        nn.init.normal_(self.fc0.bias ,0.)
        nn.init.normal_(self.fc1.weight,std =0.1)
        nn.init.normal_(self.fc1.bias, 0.)


        
    def forward(self,input_ids,attention_mask):
        hid= self.Bert(input_ids,attention_mask = attention_mask)
        hid= hid[0][:,0,:]
        x = self.d0(hid)
        x = self.fc0(x)
        x = torch.tanh(x)
        x = self.d1(x)
        x = self.fc1(x)
        return x


```
This the optimizer i used and i froze the Roberta Layers and only trained the final layers. I also trained with lr 3e-5. It didn't do any help either.
```
model = Model().to('cuda')
criterion = nn.CrossEntropyLoss(reduction='mean').to('cuda')
for params in model.Bert.parameters():
     params.requires_grad = False
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)
```
Your learning rate is pretty high and you might be overfitting. Try it again with 3E-5.<|||||>My question is i have freezed the whole roberta layer. Then why still it needs to be such small lr rather than normal 0.001 as i am training only a classifier head.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
