Hi, all:
when I train the network with such structure:

x1=F.relu(BN1(Conv2d(x)))
x1_connect=F.relu(BN1_connect(Conv2d_connect(x)))
x2=torch.cat((x1,x1_connect),dim=1)
…

Above structure is identical to:

x1=BN1(Conv2d(x))
x1_connect=BN1_connect(Conv2d_connect(x))
x2=torch.cat((x1,x1_connect),dim=1)
x2=F.relu(x2)
…

But they have totally different training results.
I have print their grad after first iteration with same initialization, input and other setting,
there grad are different in this structure.
So what could cause such problem?
Thank you!