I changed the beta values in Adam and was able to get this

validation does continue to increase passed 100 epochs albeit slowly… training continues to drop slowly
I think also the model might be extremely sensitive to learning rate… this is 3e-5
lowering it starts the causes stranger behavior such as starting losses around 0.5/0.49
I attempted cyclical learning rate but it increases the execution time by hours
edit: