I looked where this rabbit hole leads and it is way deeper, up to libcudnn.so
  

      github.com
  

  
    pytorch/pytorch/blob/master/torch/csrc/cuda/shared/cudnn.cpp


      // The clang-tidy job seems to complain that it can't find cudnn.h without this.
// This file should only be compiled if this condition holds, so it should be safe.
#if defined(USE_CUDNN) || defined(USE_ROCM)
#include <torch/csrc/utils/pybind.h>

#include <array>
#include <tuple>

namespace {
using version_tuple = std::tuple<size_t, size_t, size_t>;
}

#ifdef USE_CUDNN
#include <cudnn.h>

namespace {

version_tuple getCompileVersion() {
  return version_tuple(CUDNN_MAJOR, CUDNN_MINOR, CUDNN_PATCHLEVEL);
}




  This file has been truncated. show original

  

  
    
    
  

  


maybe someone from pytorch team will help here, but I think that It is much easier to trace where your calculations divert. For example, when you run single block with relu, have you checked that all positive values of result tensors match? Since, you know, ReLU only zeroing up negative values, so your positive values should match.