Okay, what I’m trying to do is use an existing pytorch Model (BERT) in my own custom network.
If you didn’t know, BERT works on input that is of shape (512).
As my input data is much larger than that, I’ve split each example into lengths of 12, with at most 4 lengths for each example. So for example, in my custom network, with a batch size of 32, the input shape would be (32, 4, 512).
So I’ve included a pretrained bert model in my network in the first layer, and in the forward function I loop over each of the four 512-length chunks, pass them into bert, get the output back, and stack the output back together. The output of a single bert pass is 768, therefore after the looping I now have a batch shape of (32, 4, 768).
What I’m struggling with is that my model simply isn’t converging, could it be that the backwards pass isn’t propagating to the Bert loop properly?