It might be reasonable depending on the size of all intermediate activations and the gradients.
While some layers have few parameters (e.g. conv), their activations might be large, which takes a lot of memory.
To make sure your code doesn’t have some bugs, you could post some code snippets if that’s possible.