Only 1 layer, the hidden size is 4800, that’s the one I can’t reduce (actually I tried to go to 2400 - same thing), seq_length = 36.
Yes, floats.
The problem is caused by the other pre-trained model - the moment I take it out, I can pass even larger vector (let’s say random or generated by some other Embedding), but I need both of them in one.
I tried to use pre-trained model to pre-encode everything, but this would take about 2,000 hours, which is not feasible… so they have to sit together