Answers.Body: <p>The articles you mention likely meant <em>pretrained</em> resnet50. It can get to 85%+ accuracy in 5 epochs with Adam and 1e-3 learning rate indeed. You'd need to replace the last layer or use the timm wrapper to utilize this though.</p>
  <p>However, resnet50 is not a perfect choice for images this tiny. Something like resnet18 or even smaller would perform better and faster with no pretraining.</p>
  <p>On a minor note, you probably meant <code>enumerate(tqdm(trainloader))</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-17T12:30:43.970Z
Answers.Text: enumerate(tqdm(trainloader)) - you are right thank you! And yeah, I've use pertained resnet which gave me higher accuracy, btw how do you know that resnet50 is not suitable for 'tiny' images in cifar(I assume tiny you meant low resolution)?
Answers.UserDisplayName: qvuer7
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-16T14:46:36.217Z
Answers.LastActivityDate: 2022-09-16T14:46:36.217Z
Answers.UserDisplayName: dx2-66
Body: <p>I've been trying to construct resnet50 architecture from scratch using pytorch for classification. After construction I've run training job on CIFAR10 torchvision dataset, in 20 epochs with lr of 0.01 Ive achieved 62% accuracy on validation dataset. Afterwards I've run training job using torchvision.models.resnet50 model on the same dataset which results in slight increase in accuracy, but after searching I've found multiple articles saying that resnet50 achieving accuracy of 90+% on fifth training epoch.<br />
  Training code</p>
  <pre><code>for i in range(num_epochs):
    train_loss = 0
    correct = 0
    total = 0
    
    net.train()
    for j, (image, label) in tqdm(enumerate(trainloader)):
      image = image.to(device)
      label = label.to(device)
      optimizer.zero_grad()
      out = net(image)
      loss = criterion(out, label)
      loss.backward()
      optimizer.step()
      train_loss += loss.item()
      _, predicted = out.max(1)
      total +=  label.size(0)
      correct += predicted.eq(label).sum().item()
    print('Loss: %.3f | Acc: %.3f%% (%d/%d)'% (train_loss/(j+1), 100.*correct/total, correct, total))
  
    test_loss = 0
    total = 0
    correct = 0
    net.eval()
    for l, (image, label) in tqdm(enumerate(testloader)):
      image = image.to(device)
      label = label.to(device)
      out = net(image)
      loss = criterion(out, label)
  
      test_loss += loss.item()
      _, predicted = out.max(1)
      total += label.size(0)
      correct += predicted.eq(label).sum().item()
    print('Loss: %.3f | Acc: %.3f%% (%d/%d)'% (test_loss/(l+1), 100.*correct/total, correct, total))
      
  </code></pre>
  <p>Model code, optimiser and criterion codes:</p>
  <pre><code>net = models.resnet50(num_classes = 10, pretrained = False)
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate,momentum=0.9, weight_decay=5e-4)
  </code></pre>
  <p>Datasetloading and transformation code:</p>
  <pre><code>    transform = transforms.Compose(
      [transforms.ToTensor(),
       transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
  
  
  
  trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                          download=True, transform=transform)
  trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                            shuffle=True, num_workers=2)
  
  testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                         download=True, transform=transform)
  testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                           shuffle=False, num_workers=2)
  
  classes = ('plane', 'car', 'bird', 'cat',
             'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
  </code></pre>
  <p>If OHE of dataset labels could be the case? if not please guide be what's could cause low accuracy results</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-15T19:56:13.610Z
Comments.Text: It would be also very informative, both for you and us if you could present learning curves, this could be helpful to estimate under or over fitting.
Comments.UserDisplayName: GKozinski
Tags: <pytorch><residual-networks>
Title: Whats wrong with my resnet50 training on CIFAR10 pytorch?
UserDisplayName: qvuer7
ThreadText: # Whats wrong with my resnet50 training on CIFAR10 pytorch?
  
  (Asked by: username_0 on 2022-09-15)
  
  I've been trying to construct resnet50 architecture from scratch using pytorch for classification. After construction I've run training job on CIFAR10 torchvision dataset, in 20 epochs with lr of 0.01 Ive achieved 62% accuracy on validation dataset. Afterwards I've run training job using torchvision.models.resnet50 model on the same dataset which results in slight increase in accuracy, but after searching I've found multiple articles saying that resnet50 achieving accuracy of 90+% on fifth training epoch.
  Training code
  for i in range(num_epochs):
    train_loss = 0
    correct = 0
    total = 0
    
    net.train()
    for j, (image, label) in tqdm(enumerate(trainloader)):
      image = image.to(device)
      label = label.to(device)
      optimizer.zero_grad()
      out = net(image)
      loss = criterion(out, label)
      loss.backward()
      optimizer.step()
      train_loss += loss.item()
      _, predicted = out.max(1)
      total +=  label.size(0)
      correct += predicted.eq(label).sum().item()
    print('Loss: %.3f | Acc: %.3f%% (%d/%d)'% (train_loss/(j+1), 100.*correct/total, correct, total))
  
    test_loss = 0
    total = 0
    correct = 0
    net.eval()
    for l, (image, label) in tqdm(enumerate(testloader)):
      image = image.to(device)
      label = label.to(device)
      out = net(image)
      loss = criterion(out, label)
  
      test_loss += loss.item()
      _, predicted = out.max(1)
      total += label.size(0)
      correct += predicted.eq(label).sum().item()
    print('Loss: %.3f | Acc: %.3f%% (%d/%d)'% (test_loss/(l+1), 100.*correct/total, correct, total))
      
  
  Model code, optimiser and criterion codes:
  net = models.resnet50(num_classes = 10, pretrained = False)
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate,momentum=0.9, weight_decay=5e-4)
  
  Datasetloading and transformation code:
      transform = transforms.Compose(
      [transforms.ToTensor(),
       transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
  
  
  
  trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                          download=True, transform=transform)
  trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                            shuffle=True, num_workers=2)
  
  testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                         download=True, transform=transform)
  testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                           shuffle=False, num_workers=2)
  
  classes = ('plane', 'car', 'bird', 'cat',
             'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
  
  If OHE of dataset labels could be the case? if not please guide be what's could cause low accuracy results
  
  **Tags:** pytorch, residual-networks
  > **Comments:**
  > **username_1**: It would be also very informative, both for you and us if you could present learning curves, this could be helpful to estimate under or over fitting.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-09-16. Score: 2 | Accepted answer
  
  The articles you mention likely meant *pretrained* resnet50. It can get to 85%+ accuracy in 5 epochs with Adam and 1e-3 learning rate indeed. You'd need to replace the last layer or use the timm wrapper to utilize this though.
  
  However, resnet50 is not a perfect choice for images this tiny. Something like resnet18 or even smaller would perform better and faster with no pretraining.
  
  On a minor note, you probably meant `enumerate(tqdm(trainloader))`
  
  > **Comments:**
  > **username_0**: enumerate(tqdm(trainloader)) - you are right thank you! And yeah, I've use pertained resnet which gave me higher accuracy, btw how do you know that resnet50 is not suitable for 'tiny' images in cifar(I assume tiny you meant low resolution)?  
  
  ---
  
  