Thanks for reply,    This is my first time to train a model myself, so maybe have a few of faults, thanks for reading my code 

I have used the xavier init before, but it seems result still not change
the init code:
class GNCNN(nn.Module):
  def __init__(self):
    super(GNCNN, self).__init__()
    self.gaussian1 = gaussian1
     ... ...
    self.fc3 = nn.Linear(128, 2)
    self.reset_parameters()

		
  def forward(self, x):
    ... ...

  # model init
  def reset_parameters(self):
    for mod in self.modules():
      if isinstance(mod, nn.Conv2d):
        nn.init.xavier_uniform_(mod.weight.data)
        mod.bias.data.fill_(0.2)
    
      elif isinstance(mod, nn.Linear):
        nn.init.kaiming_normal_(mod.weight.data)
        mod.bias.data.zero_()


Today i try to change my dataset from label 0 : label 1 = 1 : 1 to label 0 :  label 1 = 2 : 1
then the result change, loss become 0.64 and acc become 67%, I think that proof my model did not be trained at all:sob: (the result always change with the change of the dataset)
Epoch 35/99
----------

Train: Epoch [35] Loss: 0.6493 Acc: 66.6667%

Epoch 36/99
----------

Train: Epoch [36] Loss: 0.6493 Acc: 66.6667%


Then i try to delete relu function, but the result is still not good.
the loss change randomly, and finally to 0.64 again(after changing my dataset)
Code:
  def forward(self, x):
    prep = F.conv2d(x, KV, padding=2)
    out = self.avg_pool1(gaussian1(self.conv1(prep)))
    out = self.avg_pool2(gaussian2(self.conv2(out)))
    out = self.avg_pool3(gaussian2(self.conv3(out)))
    out = self.avg_pool4(gaussian2(self.conv4(out)))
    out = self.avg_pool5(gaussian2(self.conv5(out)))
    out = out.view(out.size(0), -1)
    out = self.fc1(out)
    out = self.fc2(out)
    out = self.fc3(out)
    return out

result:
Epoch 2/99
Train: Epoch [2] Loss: 0.6372 Acc: 66.6667%
----------

Epoch 4/99
Train: Epoch [4] Loss: 0.8944
---------

Epoch 6/99
Train: Epoch [6] Loss: 0.6770
--------

Epoch 18/99
Train: Epoch [18] Loss: 0.6513 
--------

Epoch 30/99
Train: Epoch [30] Loss: 0.6474


THanks again