Hi,
I have a weird error that I don’t quite know what to do with. After training a model for several hours without problems, I receive the following error:

terminate called after throwing an instance of ‘std::out_of_range’
what():  vector::_M_range_check: __n (which is 1903610121) >= this->size() (which is 5)
Aborted

This error is reproducible for different users using Pytorch 1.1 or Pytorch 1.3 (my config: CUDA 10.1, cuDNN 7.5, driver 418.39, NVIDIA Tesla V100).
Not being able to make any sense of the error, I tried to find the issue by using

with torch.autograd.detect_anomaly():

Interestingly, the error disappeared. My models have been running over night without issue. So, my first question is: What kind of settings does torch.autograd.detect_anomaly() change that could affect this issue?
Of course, I would like to find a permanent solution by getting to the root of the problem. I have implemented my own autograd Function here:

  
      github.com
  
  
    pputzky/invertible_rim/blob/master/irim/core/invert_to_learn.py 4
from abc import ABC, abstractmethod
import itertools

import torch
from torch import nn
from torch.autograd import Function


class InvertToLearnFunction(Function):
    @staticmethod
    def forward(ctx, n_args, layer, mode, args, kwargs, *tensors):
        if n_args == 1:
            x = tensors[0]
        else:
            x = list(tensors[:2 * n_args:2])

        with torch.no_grad():
            if mode == 'forward':
                y = layer._forward(x, *args, **kwargs)
            elif mode == 'reverse':


  This file has been truncated. show original

  
  
    
    
  
  


I have isolated the problem so far that I can say it must either have to do with InvertToLearnFunction.backward or with InvertibleLayer.gradfun, but that’s about it. This has proven really difficult to debug as the error appears after several hours of successfully running. Any insights or suggestions would be highly appreciated.