it’s quite hard to give you a MWE because of how involved the code is. I have a conda env and script here

  
      github.com
  
  
    makslevental/merf/blob/master/figures/run_times.py 1
import csv
import os
import time
from pathlib import Path

import numpy as np
import torch
from torch.backends import cudnn
cudnn.deterministic = True
from torch.utils.data import DataLoader
from skimage import io, img_as_float
from skimage.filters import gaussian
import torch.multiprocessing as mp
from nn_dog import PIN_MEMORY, DEVICE, NUM_GPUS
from nn_dog.data import SimulPLIF
from nn_dog.model import DifferenceOfGaussiansFFT, DifferenceOfGaussiansStandardConv, DifferenceOfGaussiansFFTParallel, \
    close_pool
from sk_image.blob import cpu_blob_dog
from sk_image.enhance_contrast import stretch_composite_histogram



  This file has been truncated. show original

  
  
    
    
  
  


i suspect it’s a memory leak because if i make shorter runs (i’m iterating over a set of hyperparameters) then there is no segfault. is it possible to use something like valgrind to investigate this?
also btw @albanD setting torch.backends.cudnn.deterministic = True did not actually fix the convolution strategy (i.e. force a fixed strategy); here is what i see

edit: @Adam_Dziedzic i also can’t see which cuda conv strategy is being used after running nvprof. only thing i can see is implicit_sgemm.