Hello,
I’m training an incremental classification model (ResNet-32) on the CIFAR-100 dataset.
Using cross-entropy loss, I can get the final 43.2% accuracy. I fixed random seed for each random module (eg Python built-in random module, np.random, torch, etc.), so I can get the same accuracy each time if I trained the model many times.
However, if I put some other torch.tensor on GPU, it causes the accuracy drop (42.6%). And indeed this torch.tensor is not used in training.
(torch.tensor example, self.class_sizes = torch.tensor([20, 20, 500, 500,.....]).view(-1, 1).to(self._device), and self.class_sizes doesn’t be used in training process. I just put this variable there.)
Does anyone have any ideas or encountered similar situations? Thanks!
conda environment
pytorch=1.3.0=py3.7_cuda10.0.130_cudnn7.6.3_0
torchvision=0.2.2
cudatoolkit=10.0.130=0

0809 Update:
Another similar situation
If I have a Python class (BigModel below), which includes the training/test procedure.
class AuxModel(nn.Module):
    def __init__(self):
        super(AuxModel, self).__init__()

        self.align_FC = nn.Linear(64, 1)

        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
                if m.bias is not None:
                    m.bias.data.zero_()

    def forward(self, x):
        return self.align_FC(x)

class BigModel():
    def __init__(self, args):
           ....
           self._network = ResNet32(...)

           self.SP = AuxModel()

    def train(self, ...):
            ....

    def test(self, ...):
            ....

If I train the model (ResNet32), and totally not used self.SP, I will get lower accuracy (74.2%).
But if I directly comment out the #self.SP, I can get higher accuracy (76.6%). Another thing I can know is that I do not use the self.SP in my code, because I directly commented out that, I did not get any error message.