I tried to use gc.collect() but it does not help at all.
Below are the two architectures side-by-side and the only modification to the whole code, the left one is the original one where the problem does not exist, the one on the right is the modified one which causes the issue:

image (1)1014×327 84 KB