Besides the parameters, the activations will consume a large portion of the memory.
Are you using less pooling layers (or other setups) in your custom model, which could yield larger output activations?