I’ve tested your code for some random input and the model could fit it.
Using CIFAR10, I just set the channels to 6 and  12 in the conv layers.
The model indeed struggles to learn and the default random init seems to be better.
Changing fan_in = shape[1] for linear layers makes it a bit better.
I don’t think you have a code bug, but probably your current custom initialization does not provide the benefit you expect.