Thank you! As you suggested, I am now using BCEWithLogitsLoss and the output of my model is a linear layer without a activation function after it. It seems to work quite good.
But I have a general question. I tried two approaches:

Train only on train set and tune thresholds on the validation set
Train on train + validation set and tune thresholds on these sets as well

Is approach 2 prone to overfitting? Are there any reasons not to do this? I get slightly better results when I do it like this. Of course I have also a test split which I never see during training or threshold tuning. I evaluated both approaches on the test set and get better results with #2. This is most likely due to more training samples during training.