ptrblck:

e.g. conv layers, you would see a huge memory difference, while linear layers could yield the inverse effect.


Hmm that’s a good point because of weight sharing… Does my transformer have weight sharing (I’m using XLNet)? I think yes because of recurrence right?
Also I guess the remedy is pipeline parallelism right?