Could you check, if all prediction are biased towards one specific class?
I tried your model with some dummy inputs and get a pretty decent loss:
model = TorchNet(2, 2, 2, 2)
x = torch.randn(100, 2)
target = torch.randint(0, 2, (100,))
criterion = nn.CrossEntropyLoss()

output = model(x)
loss = criterion(output, target)
print(loss)
> tensor(0.6932, grad_fn=<NllLossBackward>)

If your loss is higher, you might want to check your initializations.