You init_hidden in every iteration, and init_hidden function will create a Variable which need to be tracked grad. I think this is the reason of the “leak”. Try to remove the init_hidden in iteration or move it out of the iteration and see the memory again.