Oh, thanks @ptrblck! This was unwanted, but unfortunally it doesn’t really behave differently after changing that.
What I don’t understand is, that the network gives me always outputs close to each other, even though the input differs (alot).
For example this input:
net1 = H_0(7,3,2)
net2 = H_0(7,3,2)

x_train = torch.tensor([[-500.0,-600.0,-340,0,-900000,-200,-66]])
x_train1 = torch.ones(1,7)
x_train2 = torch.rand(1,7)

print(net1(x_train))
print(net1(x_train1))
print(net1(x_train2))

print(net2(x_train))
print(net2(x_train1))
print(net2(x_train2))

results in the following output:
tensor([[-0.0762858838]], grad_fn=<AddmmBackward>)
tensor([[-0.0739296228]], grad_fn=<AddmmBackward>)
tensor([[-0.0765117779]], grad_fn=<AddmmBackward>)
tensor([[0.4121777117]], grad_fn=<AddmmBackward>)
tensor([[0.3294166028]], grad_fn=<AddmmBackward>)
tensor([[0.3289835751]], grad_fn=<AddmmBackward>)

For each network the output is kind of constant while the input differs.
I know, that the sigmoid squashes into the interval [0,1] but shouldn’t it differ a bit more?
And if I use the H_0 networks in my bigger network recursively I get outputs that converge to a constant because each H_0 behaves like that.
By the way this network is motivated by a theoretical result for rates of convergence in nonparametric regression using feedforward neural networks by Kohler and Bauer (2019)
Bauer, Benedikt; Kohler, Michael. On deep learning as a remedy for the curse of dimensionality in nonparametric regression.
Ann. Statist. 47 (2019), no. 4, 2261–2285. doi:10.1214/18-AOS1747. https://projecteuclid.org/euclid.aos/1558425645