Are you sure the depth images are not stored in 16bit?
Anyway, are you training your model from scratch or finetuning a pretrained model?
Was the accuracy better when you loaded the depth images as 3-channel images?