I don’t think the issue is in the model, but still the input.
In your current code you are checking for 4 channels and are only concatenating the image tensors then:
if self.input_channels == 4:
    ...
    image = np.concatenate((rgb_image, dep_image), axis=-1)

which seems to be wrong. Maybe you want to check for 3 channels to concatenate the images to 4 channels?