Solved by ptrblck in post #2 
              
              
                You could use self.register_buffer and set the persistent argument to False. 
From the docs: 

persistent (bool) – whether the buffer is part of this module’s state_dict.