I changed how I was saving the model in python to it being saved in a CPU state, that way it wouldn’t be GPU dependent in the C++.
m = model.module
m.to(cpu_device)  # move to cpu
traced = torch.jit.trace(m, example, check_trace=False)
torch.jit.save(traced, "saved_models/test_model.pt") 
m.to(device) # move back to gpu

This seems to not affect the training of my model, and actually does serve as a workaround for the ASAN issue. However, I still think the memory leak is something to be looked at if it was previously unknown?