If you are using an older version of pytorch I think I remember there being an issue with that so upgrading pytorch newest version would help. If not and in addition to that when you interrupt training use Ctrl C to end your training and that should kill the processes properly and free up the gpu memory