While I agree with those above arguing that train mode validation loss calculation is fine, there still a serious efficiency problem here.
If you also want the model outputs (for tracking IOU, accuracy, etc., which is often the case), then you need to run inference twice.  Training mode for the loss, and eval mode for the outputs. Would be better to have both with a single pass!