What memory usage do you expect and how are you measuring it?
Calculating the tensor shapes of all parameters and buffers as well as the intermediate activations and gradients would give you an approx. estimate of the memory footprint.
Also, you could use the memory profiler to check where memory is allocated.