512 is the number of channels, i.e. also the number of linear layers?
The for loop might slow down the linear approach, so I guess the conv approach would be faster, but to be sure I would just time them in a small function using random input.
Remember to use torch.cuda.synchronize() before starting and stopping the timer, if you want to compare both approaches on the GPU.