I agree with you for not using sigmoid.
I have since changed the code. Coming from Keras, I’m used to putting activations as part of the main model. Since the loss function used in YOLO is a combination of multiple things, with classification objective and regression objectives, I should not have used a sigmoid layer on the network. Instead, I have removed the sigmoid, and applied different activations for different parts of the output.