Yes the codebase is relatively big and impossible to post to a forum unfortuantely.
I suspect you may be right though. Replacing the global (yes in main() function) autogradmode(false) with a grad guard removes the memory accumulation as well. Do you have an idea how to best track down a seemingly unknown call to torch?
Edit:
I figured out where the hidden call to torch actually comes from. A clear fault on my end. Thank you for your help!