Small update: I did try a couple of loss functions (MSE with mod 2pi, atan2) but nothing surprised me. I had another idea of doing multi-output classification. Actually my 512 phases at the end on my dataset do come on 128 discretized levels (because of hardware limitation issues, aliasing etc.) and I could take advantage of that. Instead of having only one fork (fully connected layer) at the end I could have 512 small networks, each of them having 128 outputs with a sigmoid activation function, and train on nn.CrossEntropyLoss.