Answers.Body: <p>Every training will be slightly different because of the statistical matter of neural networks.</p>
  <p>The question is, how much is your better?</p>
  <p>Then, newer does not imperatively mean better. It means more hardware and computational power, but not necessarily how it is used for specific tasks.</p>
  <p>Maybe, the 3090 can't leverage its power in this specif task properly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-20T14:05:23.957Z
Answers.Text: But isn't image classification with CNN model a pretty standard task in DL? From your answer I reckon that if a person or a company wants to increase their computational power to speed up their work, they will have to buy couple of different GPUs and test each of their different task's models on each of these? Is this really how it works? Am I suppose to have 3 different PCs and for each task always find the best one? Buying card that has more computational power isn't enough anymore?
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-20T18:56:08.640Z
Answers.Text: @GKozinski I can't really say more. I have no insight in your set-up etc. I just gave you some general hints. Maybe check your drivers, or the task is so trivial that the higher power of the 3090 becomes irrelevant.
Answers.UserDisplayName: Skobo Do
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-19T09:46:41.060Z
Answers.LastActivityDate: 2022-10-19T09:46:41.060Z
Answers.UserDisplayName: Skobo Do
Body: <p>Can someone explain why training CNN model (in my case DenseNet201) on the same data, and the same data processing pipeline can be slower on better GPU (RTX3090) than worse one (RTX3060), with the same other parameters (exactly same PC just with new GPU)?</p>
  <p>In both cases I used same batch size and other settings. The only way to make training faster on 3090 was to actually increase the batch size, which was too big for 3060. But I still don't understand why the same training params wouldn't produce better results.</p>
  <p>Even though big part of the training is reading data from disk and data augmentation (albumentations in my case) it's still the same setup, so even if GPU work is smaller part of one entire epoch, it still should be a bit faster, right?</p>
  
Tags: <deep-learning><convolutional-neural-networks><training><gpu><hardware>
Title: Why training the same model on the same data can be slower on better card?
UserDisplayName: GKozinski
ThreadText: # Why training the same model on the same data can be slower on better card?
  
  (Asked by: username_0 on 2022-10-18)
  
  Can someone explain why training CNN model (in my case DenseNet201) on the same data, and the same data processing pipeline can be slower on better GPU (RTX3090) than worse one (RTX3060), with the same other parameters (exactly same PC just with new GPU)?
  
  In both cases I used same batch size and other settings. The only way to make training faster on 3090 was to actually increase the batch size, which was too big for 3060. But I still don't understand why the same training params wouldn't produce better results.
  
  Even though big part of the training is reading data from disk and data augmentation (albumentations in my case) it's still the same setup, so even if GPU work is smaller part of one entire epoch, it still should be a bit faster, right?
  
  **Tags:** deep-learning, convolutional-neural-networks, training, gpu, hardware
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-19. Score: 1
  
  Every training will be slightly different because of the statistical matter of neural networks.
  
  The question is, how much is your better?
  
  Then, newer does not imperatively mean better. It means more hardware and computational power, but not necessarily how it is used for specific tasks.
  
  Maybe, the 3090 can't leverage its power in this specif task properly.
  
  > **Comments:**
  > **username_0**: But isn't image classification with CNN model a pretty standard task in DL? From your answer I reckon that if a person or a company wants to increase their computational power to speed up their work, they will have to buy couple of different GPUs and test each of their different task's models on each of these? Is this really how it works? Am I suppose to have 3 different PCs and for each task always find the best one? Buying card that has more computational power isn't enough anymore?  
  > **username_1**: @username_0 I can't really say more. I have no insight in your set-up etc. I just gave you some general hints. Maybe check your drivers, or the task is so trivial that the higher power of the 3090 becomes irrelevant.  
  
  ---
  
  