Answers.Body: <p>There are a bunch of things that can influence how fast your code runs, not necessarily to do with the architecture but more programming wise. For example, you can have slow implementations of adding and processing sentence padding, for computing loss functions and who knows what more. Without taking a proper look at your code it is hard to say what takes up most of the time.</p>
  <p>I would advise you to put some <code>time.time()</code> statements in your code, and time what parts of your training loop take the majority of time (scale them to percentages and list them). Then you can open another question with trying to get general advice on speeding up such a computation, or post code on the Data Science SE for specific help.</p>
  <p>One thing you can always do is make your batch size larger which should easily make your model much faster. There are however also other consequences to changing your batch size which are dependent on your model/data etc (so i cannot specifically state what it will do for you).</p>
  <p>Also, if you do not want to do fancy weird models, and can get away with the Tensorflow Keras module, that one is usually much faster as it is more optimized than the general (torch or TF) frameworks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-18T14:15:41.003Z
Answers.Text: Rather than hand-decorating code with `time.time()` statements, a more robust approach is to use a profiler. See https://docs.python.org/3/library/profile.html
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-18T14:57:24.660Z
Answers.Text: Thank you for your answer.
  I timed my model and uploaded it as a question on DS SE: https://datascience.stackexchange.com/questions/117858/how-can-i-make-my-neural-network-learn-faster
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-18T13:12:00.927Z
Answers.LastActivityDate: 2023-01-18T13:12:00.927Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am using a GPU provided by Google called A100-SXM4-40GB, and my framework of choice is Pytorch. With my current training speed, I get through about 650,000 training examples/sentences in 12 hours. My vocabulary size is about 85,000, the number of parameters in the VAE is 17.1 million (mostly high because of the embedding layer). My batch size is 64, and I am using the Adam optimizer.</p>
  <p>What advice can you give me to speed up my model? For instance, I also have access to a TPU, but I have never seen a clear breakdown on GPU vs TPU performance (and what role batch size plays). Can I use parallel computing, and is this possible on Colab? I know I might have to make some concessions with the training set size, but I would like to get through as much data as possible in 48 hours of training.</p>
  
Tags: <optimization><pytorch><gpu>
Title: How to speed up my neural network?
UserDisplayName: postnubilaphoebus
ThreadText: # How to speed up my neural network?
  
  (Asked by: username_0 on 2023-01-17)
  
  I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am using a GPU provided by Google called A100-SXM4-40GB, and my framework of choice is Pytorch. With my current training speed, I get through about 650,000 training examples/sentences in 12 hours. My vocabulary size is about 85,000, the number of parameters in the VAE is 17.1 million (mostly high because of the embedding layer). My batch size is 64, and I am using the Adam optimizer.
  
  What advice can you give me to speed up my model? For instance, I also have access to a TPU, but I have never seen a clear breakdown on GPU vs TPU performance (and what role batch size plays). Can I use parallel computing, and is this possible on Colab? I know I might have to make some concessions with the training set size, but I would like to get through as much data as possible in 48 hours of training.
  
  **Tags:** optimization, pytorch, gpu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-18. Score: 2 | Accepted answer
  
  There are a bunch of things that can influence how fast your code runs, not necessarily to do with the architecture but more programming wise. For example, you can have slow implementations of adding and processing sentence padding, for computing loss functions and who knows what more. Without taking a proper look at your code it is hard to say what takes up most of the time.
  
  I would advise you to put some `time.time()` statements in your code, and time what parts of your training loop take the majority of time (scale them to percentages and list them). Then you can open another question with trying to get general advice on speeding up such a computation, or post code on the Data Science SE for specific help.
  
  One thing you can always do is make your batch size larger which should easily make your model much faster. There are however also other consequences to changing your batch size which are dependent on your model/data etc (so i cannot specifically state what it will do for you).
  
  Also, if you do not want to do fancy weird models, and can get away with the Tensorflow Keras module, that one is usually much faster as it is more optimized than the general (torch or TF) frameworks.
  
  > **Comments:**
  > **username_2**: Rather than hand-decorating code with `time.time()` statements, a more robust approach is to use a profiler. See https://docs.python.org/3/library/profile.html  
  > **username_0**: Thank you for your answer. I timed my model and uploaded it as a question on DS SE: https://datascience.stackexchange.com/questions/117858/how-can-i-make-my-neural-network-learn-faster  
  
  ---
  
  