I am working on the DCASE 2016 6 challenge acoustic scene classification problem using CNN. All training data (audio files .wav) are converted into a size of 1024x1024 JPEG of MFCC output.

EPOCH = 10
BATCH_SIZE = 30
LR = 0.001
   self.layer1 = nn.Sequential(                     # 1024x1024x3
       nn.Conv2d(3,96,13,3,3),                      # 340x340x96
       nn.ReLU(),
       nn.MaxPool2d(2),                             # 170x170x96
       nn.Conv2d(96,256,10,2,1),                    # 82x82x256
       nn.ReLU(),
       nn.MaxPool2d(2),                             # 41x41x256
       nn.Conv2d(256,384,5,2,1),                    # 20x20x384
       nn.ReLU(),
       nn.Conv2d(384,384,3,1,1),                    # 20x20x384
       nn.ReLU(),
       nn.Conv2d(384,256,3,1,1),                    # 20x20x256
       nn.MaxPool2d(2))                            # 10x10x256
   self.fc = nn.Linear(10*10*256, 15)


However, with this configuration the loss never decreases but fluctuating throughout the entire run. And the final accuracy will always stuck at 6%. Can anyone help in guiding me? Thank you.