Hi Peter,
@peterjc123
the function takes the image (3d numpy array) and get a subvolume according to the key I pass to the function. I also do some preprocessing (removing the unconnected objects of the image using a python package).
Regardless, it is not clear to me what is the best practices for using multiprocessing if I want to load 3d numpy arrays (646464) for training, with their labels. I have around 17000 data points for training. I read the multiprocessing best practices in pytorch documentation but I did not get much that would give an indictation to the fastest way for loading such data. any suggestions
def shift(img, Bs, sh, key, direction):
   "get a subvolume from the bigger image according to the given key"
   "Bs is the subvolume size in voxels default=64"
   "sh is the shift in voxels at each step similar to stride in deep learning"
   "key is the subvolume Id"

   key = key - 1
   "transpose 3D matrix according to the direction of Flow"

   if direction == 'Z':
       pass
   if direction == 'X':
       img = np.transpose(img, [2, 1, 0])
   if direction == 'Y':
       img = np.transpose(img, [0, 2, 1])

   # img[img == 0] = 2
   Nx = img.shape[0]
   Ny = img.shape[1]
   Nz = img.shape[2]

   Nbx = np.floor((Nx - Bs) / sh).astype(int)
   Nby = np.floor((Ny - Bs) / sh).astype(int)
   Nbz = np.floor((Nz - Bs) / sh).astype(int)

   [i, j, k] = np.unravel_index(key, [Nbx + 1, Nby + 1, Nbz + 1], 'F')

   starti = sh * i;
   endi = sh * i + Bs;
   startj = sh * j;
   endj = sh * j + Bs;
   startk = sh * k;
   endk = sh * k + Bs;

   cube = img[starti:endi, startj:endj, startk:endk]

   # Remove unconnected parts
   labels_inv = util.invert(cube)
   labels_in = labels_inv.astype(np.int32)

   connectivity = 6  # only 26, 18, and 6 are allowed
   labels_out = cc3d.connected_components(labels_in, connectivity=connectivity)  # connected components labels
   N = np.max(labels_out)  # number of labels
   for segid in range(1, N + 1):  # remove smaller parts
       if labels_out[labels_out == segid].size < labels_in.shape[0]:  # 64
           labels_out[labels_out == segid] = 0
   if N > 1:
       for segid in range(1, N + 1):
           z = np.unravel_index(np.where(labels_out == segid),
                                (labels_in.shape[0], labels_in.shape[0], labels_in.shape[0]), 'F')[0][2]
           numIn = np.sum(z == 0)
           numOut = np.sum(z == labels_in.shape[2]-1)
           if numIn == 0:
               labels_out[labels_out == segid] = 0
           elif numOut == 0:
               labels_out[labels_out == segid] = 0
           else:
               labels_out[labels_out == segid] = 1

   return labels_out.astype('float64')