Actually, what does it mean when I stack 2 linear layers with nn.Sequential(nn.Linear, nn.Linear) in PyTorch? Wouldn’t that make the learning already non-linear?
If I stack 2 linear layers with sigmoid activation, wouldn’t it make it non-linear? Or is it only specific to PyTorch where two linear layers are somehow smashed into one and learn a single linear function to map the
input to the output without a hidden layer?
If we look at https://www.kaggle.com/alvations/xor-with-mlp 22 (at the end of the notebook, without pytorch), simply stacking two layers of perceptrons and doing a non-absolute L1 loss would allow non-linearity to set in.  And the network would have learnt to represent the XOR function.
Also, written by hand, the math checked out for the two layered perceptrons: http://www.aclweb.org/anthology/S16-1148 17
Wouldn’t the non-linearly come directly from the fully connected layers between the hidden dimension and the in-/output? There shouldn’t be a need to force a non-linear activation between the hidden layer.