I am very happy with this solution.
As far as I can tell the most important problems were wrong activation/loss functions. After these changes many nets seem to work, but your small net does the trick fast and very accurate. I am also very happy you showed an example of the lr_scheduler. I saw them a few days ago but forgot about them again.
However I do not understand your point about target bias. What exactly is target bias and how do you come up with the exact number of 25?
Edit: I just noticed very strange behaviour. I ran my code with all your chnges a few times and it always looks like the loss reaches a plateau at around 1 and stays there for a long time, then suddenly it gets much better. The length of this plateau varies between 200 and 800 epochs in my tests. Do you have an idea why this is? Especially when it stayed really constant at loss 1 for 800 epochs I surely would have stopped the net if it wasn’t so fast.