Is your model on the GPU or CPU? If switching to the other one doesn’t OOM then there’s probably a memory leak somewhere.
In general my advice is to try to narrow down where the memory leak is by deleting code until the memory stops leaking.