It sounds crazy that I’m answering my own question but I’m just going to write these down so that it’s somehow documented here and I can come back to check when in doubt =)
It’s because the second nn.init is not re-initializing all the m.weight but only the 0th row.

>>> m = nn.Embedding(10, 128, padding_idx=0)
>>> nn.init.uniform_(m.weight, -0.1, 0.1)
>>> print(m.weight)
Parameter containing:
tensor([[ 0.0270, -0.0801, -0.0744,  ...,  0.0148, -0.0218,  0.0371],
        [-0.0279,  0.0053, -0.0801,  ..., -0.0808, -0.0775, -0.0291],
        [-0.0704, -0.0537, -0.0336,  ...,  0.0730,  0.0389,  0.0482],
        ...,
        [-0.0324, -0.0685, -0.0629,  ..., -0.0472, -0.0990, -0.0958],
        [ 0.0852,  0.0468,  0.0605,  ..., -0.0059,  0.0389, -0.0629],
        [-0.0258,  0.0022,  0.0793,  ...,  0.0473,  0.0635, -0.0056]],
       requires_grad=True)

>>> nn.init.constant_(m.weight[0], 0)
>>> print(m.weight)
Parameter containing:
tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [-0.0279,  0.0053, -0.0801,  ..., -0.0808, -0.0775, -0.0291],
        [-0.0704, -0.0537, -0.0336,  ...,  0.0730,  0.0389,  0.0482],
        ...,
        [-0.0324, -0.0685, -0.0629,  ..., -0.0472, -0.0990, -0.0958],
        [ 0.0852,  0.0468,  0.0605,  ..., -0.0059,  0.0389, -0.0629],
        [-0.0258,  0.0022,  0.0793,  ...,  0.0473,  0.0635, -0.0056]],
       requires_grad=True)