Hello All,
I am using ResNet3D architecture with some changes:
    x = self.lastConv2(x)
    x = self.relu(x)
    x = self.lastConv1(x)
    x = self.relu(x)
    x = self.lastConv0(x)
    x = self.relu(x)
    x = x.view(x.size(0), -1)
    x = self.dropout(x) #20%
    x = self.fc(x)
    x = torch.tanh(x)
    return x #torch.tanh(x)

I am facing a strange issue like the output of the FC layer is increasing in every epoch which is increasing the train MSE loss.

image812×997 149 KB

And the output of tanh ultimately becomes very clipped at +1 to -1 which I understand that tanh will be clipped at +1 and -1 since the inputs are very high.
I am using Xavier weight initialization and all the inputs are normalized from -1 to +1 scale.
def weights_init(m):
    if isinstance(m, nn.modules.conv._ConvNd): #nn.Conv3d
        init.xavier_uniform_(m.weight.data, gain = np.sqrt(2.0))
        # m.bias.data.fill_(0) #todo: if bias = False then comment out the line
        # torch.nn.init.xavier_uniform_(m.bias.data)
    elif isinstance(m, nn.modules.batchnorm._BatchNorm):
        # print('OK')
        m.weight.data.normal_(mean=1.0, std=0.02)
        # m.bias.data.fill_(0) #todo: if bias = False then comment out the line
    elif isinstance(m, nn.Linear):
        # m.weight.data.normal_(0.0, 0.02)
        # init.xavier_uniform_(m.weight.data)
        y = 1/np.sqrt(m.in_features)
        m.weight.data.uniform_(-y, y)
        m.bias.data.fill_(0) #0.01

And for the network and loss:
    Net = ResNet(Bottleneck, [1, 1, 1, 1]).to(device)
    Net.apply(weights_init)
    Optimizer = optim.Adam(Net.parameters(), lr=0.0005)
    Criterion = nn.MSELoss()
    Criterion = Criterion.cuda()

And about normalizing:
from skimage.exposure import rescale_intensity
MRimage = rescale_intensity(MRimage, in_range=(MRimage.min(), MRimage.max()),out_range=(-1,1))
target = rescale_intensity(target, in_range=(target.min(), target.max()), out_range=(-1,1))

What could be happening inside??