Answers.Body: <p>Welcome to the AI StackExchange!</p>
  <p>If you have never implemented a deep learning paper from scratch, it might be difficult to begin with a specific paper of your choice as there will not be many other implementations to help you out or other people who know the paper in-depth.</p>
  <p>I would advise you to start by implementing a very popular/widely cited deep learning paper. If you get stuck along the way, you can google other people's implementations to get some guidance or ask additional questions on the Data Science Stack Exchange.</p>
  <p>Now, to give some guidance on where to start, I'll give you <strong>my usual process</strong> of implementing papers. (Note: this is not necessarily the best or the only way, nor the way I always go about it, just the best way I could summarize my approach)):</p>
  <ol>
  <li>Read the paper on a surface level. Try to understand the key concepts (not the small details).
  <ul>
  <li>What is the problem they want to solve?</li>
  <li>What is their approach to solving the problem?</li>
  <li>What are the key deep learning concepts they are using?</li>
  <li>Do you understand the general structure of the architecture/method they propose?</li>
  <li>What data do they use? What is the format of this data?</li>
  <li>(for neural net architectures)
  <ul>
  <li>What is the loss function?</li>
  <li>Why is this loss function used?</li>
  </ul>
  </li>
  </ul>
  </li>
  <li>Dive deeper into the stuff you don't yet completely understand. Depending on what you don't understand yet, use the citations in the papers or make use of the internet (blog-posts, Stack Overflow, etc.). Understanding the concepts used in the paper is important!</li>
  <li>Start implementing the DataLoader. Get the data in the right format (see the question on data format above). Check the paper on batch size, shuffling and other specifics (if mentioned). If there are multiple data sources, take the easiest and simplest data. Always start simple!</li>
  <li>Start implementing the architecture components. Use a batch of data to continuously test whether your code works and keep extending onwards.</li>
  <li>Implement the training process. Loop over your data, train the model using the modules just implemented and calculate the loss and backpropagate. Don't forget to reset the gradients of the optimizer at the start.</li>
  <li>Test your code, and see if it runs.</li>
  <li>Go back to the paper to read all the specifics. The type of optimizer, the number of neural network layers, whether they use biases, all of it! Implement all these specific details.</li>
  <li>Test and debug time! Test your model on the training data and see if it converges in a proper manner. Debug anything that doesn't work!</li>
  </ol>
  <p>Remember, implementing deep learning papers from scratch is not an easy thing to do! Don't get frustrated if it doesn't work immediately. I can promise you it won't. Therefore, again, make sure to start with a well-known paper. Also during implementing, start easy and work your way up. Don't just start with a super deep network, with millions of parameters. Start easy.</p>
  <p>Good luck and have fun ;)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-01T09:56:55.227Z
Answers.Text: @Robin van Hoorn  thank you for the insight. I think this gives me basic idea of how to start in first place. This was a really nice answer.
Answers.UserDisplayName: programmer_04_03
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-02T10:03:08.043Z
Answers.Text: @ShubhamDeshpnde if the question has sufficiently answered your question, please upvote it and select it as the 'correct' answer. This helps future visitors of the website to quickly find what they are looking for.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-02T10:36:38.973Z
Answers.Text: @user253751 Thanks for the note, you are completely correct. I edited my answer to incorporate your suggestion.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-27T10:11:42.540Z
Answers.LastActivityDate: 2023-04-02T10:36:57.100Z
Answers.LastEditDate: 2023-04-02T10:36:57.100Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I am fairly new to deep learning. I want to implement a deep learning paper from scratch with proper data preprocessing, model, losses etc., using an object-oriented approach in python. I want to do something similar to the official implementations I see on GitHub and other websites.</p>
  <p>I have no idea where to start for this task. I will appreciate some guidance regarding this.</p>
  <p>The framework I want to use is Pytorch (Since it is more pythonic way of doing things)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-28T07:17:05.607Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Anshuman Kumar
Tags: <deep-learning><python><pytorch>
Title: How do I start implementing a deep learning paper from scratch?
UserDisplayName: programmer_04_03
ThreadText: # How do I start implementing a deep learning paper from scratch?
  
  (Asked by: username_0 on 2023-03-27)
  
  I am fairly new to deep learning. I want to implement a deep learning paper from scratch with proper data preprocessing, model, losses etc., using an object-oriented approach in python. I want to do something similar to the official implementations I see on GitHub and other websites.
  
  I have no idea where to start for this task. I will appreciate some guidance regarding this.
  
  The framework I want to use is Pytorch (Since it is more pythonic way of doing things)
  
  **Tags:** deep-learning, python, pytorch
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-27. Score: 2 | Accepted answer
  
  Welcome to the AI StackExchange!
  
  If you have never implemented a deep learning paper from scratch, it might be difficult to begin with a specific paper of your choice as there will not be many other implementations to help you out or other people who know the paper in-depth.
  
  I would advise you to start by implementing a very popular/widely cited deep learning paper. If you get stuck along the way, you can google other people's implementations to get some guidance or ask additional questions on the Data Science Stack Exchange.
  
  Now, to give some guidance on where to start, I'll give you **my usual process** of implementing papers. (Note: this is not necessarily the best or the only way, nor the way I always go about it, just the best way I could summarize my approach)):
  
  1. Read the paper on a surface level. Try to understand the key concepts (not the small details).
     * What is the problem they want to solve?
     * What is their approach to solving the problem?
     * What are the key deep learning concepts they are using?
     * Do you understand the general structure of the architecture/method they propose?
     * What data do they use? What is the format of this data?
     * (for neural net architectures)
       + What is the loss function?
       + Why is this loss function used?
  2. Dive deeper into the stuff you don't yet completely understand. Depending on what you don't understand yet, use the citations in the papers or make use of the internet (blog-posts, Stack Overflow, etc.). Understanding the concepts used in the paper is important!
  3. Start implementing the DataLoader. Get the data in the right format (see the question on data format above). Check the paper on batch size, shuffling and other specifics (if mentioned). If there are multiple data sources, take the easiest and simplest data. Always start simple!
  4. Start implementing the architecture components. Use a batch of data to continuously test whether your code works and keep extending onwards.
  5. Implement the training process. Loop over your data, train the model using the modules just implemented and calculate the loss and backpropagate. Don't forget to reset the gradients of the optimizer at the start.
  6. Test your code, and see if it runs.
  7. Go back to the paper to read all the specifics. The type of optimizer, the number of neural network layers, whether they use biases, all of it! Implement all these specific details.
  8. Test and debug time! Test your model on the training data and see if it converges in a proper manner. Debug anything that doesn't work!
  
  Remember, implementing deep learning papers from scratch is not an easy thing to do! Don't get frustrated if it doesn't work immediately. I can promise you it won't. Therefore, again, make sure to start with a well-known paper. Also during implementing, start easy and work your way up. Don't just start with a super deep network, with millions of parameters. Start easy.
  
  Good luck and have fun ;)
  
  > **Comments:**
  > **username_0**: @username_2  thank you for the insight. I think this gives me basic idea of how to start in first place. This was a really nice answer.  
  > **username_2**: @ShubhamDeshpnde if the question has sufficiently answered your question, please upvote it and select it as the 'correct' answer. This helps future visitors of the website to quickly find what they are looking for.  
  > **username_2**: @user253751 Thanks for the note, you are completely correct. I edited my answer to incorporate your suggestion.  
  
  ---
  
  