I guess you might observe memory fragmentation, i.e. the increased model size does not fit in the old memory block reserved for the smaller model, so a new one might be allocated.
Would it work if you start using the biggest model and shrink it?
I haven’t tested it yet, but it might be worth a try.