I tried a few approaches using different weight init methods etc., but the main issue seems to be the loss in numerical precision using sigmoid + nn.BCELoss.
If you remove the sigmoid in your model and use nn.BCEWithLogitsLoss as your criterion (which uses the LogSumExp trick for stability), your model seems to converge in all runs.