Based on your code snippet I cannot see anything obviously wrong, but since you are using custom sumbodules, I’m not sure if they contain any bugs.
The memory usage depends on a lot of factors besides the number of layers.
E.g. if you pool your activations quickly, you could use a lot of more layers.