i would know how to do that with tensorflow, but i dont know how i can do this is i use simple transformers to be honest. in their tutorial on binary text classification they didnt change a single layer, so i thought i would be fine leaving it like that -

  
      
      github.com
  
  
    

ThilinaRajapakse/simpletransformers - A quick example 2


  Transformers for Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conversational AI

  
  
    
    
  
  


in addition my loss is very fast converging to



i prop should use a small learning rate and just train 1 epoch to get the best result, because bert is pretrained right? is this some sort of transfer learning what im doing here? 
ill try to use a learning rate around  1.0e-5, testing small sample size, this gave me the best result - even tho its way below bilstm