So to reiterate, since I am basically changing nothing except for the way I supply embeddings, why am I observing a decreasing in performance? Are there optimizations pytorch makes on embeddings even when requires_grad is set to False?
Any help would be greatly appreciated! Thanks.