I have some updates:

If I create some tensors like: torch.zeros(bsz, max_len), will this cause some side-effect?
I output the python garbage collector, and find something wierd:

Screenshot from 2021-09-23 12-53-45816×373 46.8 KB


There is a huge dict, which contains a lot of tensors, is this part of the graph or something?