So just to clarify, If there is an activation layer(relu) in between then there is no need to turn off the bias in layer one. Right.
I have this doubt that linear layer at the end (from 2048 > 4 output class) learns less compared with (2048 > 1024 > 4 output class). If yes in what situation this is helpful? Why most people choose model2 like architecture during fine-tuning?