Solved by ptrblck in post #2 
              
              
                I guess appending to self.skips and using it afterwards could create the issue. 
Could you re-initialize self.skips = [] inside the forward method for the sake of debugging and check, if this would be working?