This is my experimental network. The first layer and the second layer is fully connected.
{first layer (linear activated)}
{second layer (linear activated)}
{the production of the neurons of the second layer}
I think the network is non-linear because of the production. The number of the last layer’s neurons increases, the non-linear ability of the network increases.
The first layer has 100 neurons and the second layer has 64 neurons. The input is an image with 8x8 pixels, each pixel is either 0 or 1.  After standard normal initialization, the output is huge.
I tried uniform initialization, the output is also huge, but the outputs’ variation is smaller than that by normal initialization.
Why am I using such structure of network is because I want to express a physics wave-function of many-body system by neural network. I tried MLP and CNN however they don’t work well,  hence I designed such kind of network.