Your example isn’t coherent with your explanation above. Given your explanation, seg and F size should be respectively (2, 2, 2) and (2, 3, 2, 2):
seg = [[[0, 1],
        [2, 3]],
       [[1, 2],
        [1, 2]]]

mask = np.array(seg)

print(mask.shape)  # (2, 2, 2)

F = [[[[2, 3],
       [4, 3]],
      [[4, 1],
       [2, 4]],
      [[4, 0],
       [5, 1]]],
     [[[1, 2],
       [0, 4]],
      [[0, 1],
       [10, 3]],
      [[1, 2],
       [0, 5]]]]

features = np.array(F)

print(features.shape)  # (2, 3, 2, 2)

output = features[mask == 1] # yields an error because mask and features haven't the same dimension


But in that case I don’t understand what you want to do. If you have a batch of segmentation masks, and a batch of features maps, don’t you want to treat each sample from the batch independantly ? Based on your example, it seems you want the positive indices across all the batch, which is implicitely a fusion of the batched segmentation masks. Is it really what you want to do ?
For instance let’s forget this batch issue and assume you have one segmentation mask and one set of features:
mask     = torch.randint(0, 5, size=(64, 64))
features = torch.rand(25, 64, 64)

You can then use torch.masked_select:
class_mask = torch.where(mask == 1, True, False)
output = features.masked_select(class_mask)