Thank you for the suggestion!
I don’t want to gapfill with the mean, rather I want to randomly sample the gap fillings from the distribution of the other pixels, so in this example i randomly want to draw pixels from the set  {[3,1,3],[4,3,3],[2,2,1],[3,1,3],[1,1,1],[3,2,2]} to fill the three gaps ([0,0,0]) in a.
So a possible solution would be
a_filled = torch.tensor(
                         [[[3,4,2],[1,2,3],[1,3,2]],
                          [[1,3,2],[1,2,1],[1,2,2]],
                          [[3,3,1],[1,1,3],[1,2,1]]])

another one would be
a_filled = torch.tensor(
                         [[[3,4,4],[3,2,3],[1,3,1]],
                          [[1,3,3],[2,2,1],[1,2,1]],
                          [[3,3,3],[2,1,3],[1,2,1]]])