By default the model parameters will be stored as FP32, so each value will take 4 Bytes.
Note that the activation, which are created during the forward pass, will also use some memory.
The activation shape would depend on the input shape and the layer setup.
After training on one mini-batch you will be able to reuse the memory and feed the next mini-batch for training. Did you see a growing memory usage?
The model is usually loaded onto the GPU before training. Each batch given from the DataLoader will be pushed to the GPU in the training loop to keep the memory usage low.