You could adapt this initialization using your non-linearities:
def weight_init(m):
    if isinstance(m, nn.Linear):
        nn.init.xavier_uniform_(m.weight, gain=nn.init.calculate_gain('relu'))
        if m.bias is not None:
            nn.init.zeros_(m.bias)

In most runs the output of the model was randomly distributed, although not in every run.