Hey,
im currently working on a classification transformer and using the imdb sentiment dataset.
And I am pretty unsure that my model is working because when i remove the transformer blocks the loss  is only 8% higher as with them.
I am using the  pretrained 100 dim glove word embeeding vectors. Thats why my hidden dim is also only 100.
model specs:
INPUT_DIM = 20000
HID_DIM = 100
OUT_DIM = 1
N_LAYERS = 2
N_HEADS = 4
FF_DIM = 32
MAXLEN = 100
optimizer = torch.optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
Google Colab Link:
  
      
      colab.research.google.com
  
  
    

Google Colaboratory 4



  
  
    
    
  
  


Can someone please tell me how to check if these attention algorithms are working?