Answers.Body: <p>Generally, your PyTorch module is to contain the architecture that you are training, not the additional variables that you want to store. So generally, I would not implement my list like this.</p>
  <p>What is usually done is to have one <code>nn.Module</code> for the architecture and one generic class for utility functions such as the model training/fitting/validation etc. Then the module is passed to the generic 'Trainer' class for training the model and your list of values would presumably be stored in the 'Trainer' object.</p>
  <p>So something like this:</p>
  <pre><code>class MyNetwork(nn.Module):
      # Here define the architecture
      def __init__(self, *other_arguments): ...
     
      def forward(batch): ...
  
  class Trainer():
      def __init__(self, network_instance, optimizer, *other_arguments):
          self.network_instance = network_instance
          self.optimizer = optimizer
  
          self.list_of_values = []
  
      def train(n_epochs):
          for i in range(n_epochs):
              # Standard train function layout
              self.optimizer.zero_grad()
              batch = self.get_data()
              predictions = self.network_instance(batch)
              loss = calculate_loss(predictions)
              loss.backward()
              self.optimizer.step()
  
              # Add value to list
              self.list_of_values.append(loss.item())
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T13:41:26.923Z
Answers.Text: I understand. It indeed makes sense to store the list in the Trainer. Now, if I store a separate model (let's say an sklearn model) in a variable of the nn.Module, and I use this accumulated list to train that model, (calling it in the trainer as `sklearn_model.fit(list_of_values)`, will those learned weights be succesfully stored inside the model variable in the nn.Module after training?
Answers.UserDisplayName: puradrogasincortar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T14:13:42.920Z
Answers.Text: I am not familiar with the integration of sklearn models (and their parameters) into PyTorch Modules. I'd advise keeping them separate if possible.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T14:42:03.960Z
Answers.Text: I actually think they are not made to work together in terms of saving sklearn model weights for instance, at least from my experiments. I guess the right choice would be to implement the sklearn model in Pytorch and then load this model as an instance of the nn.Module class. Many thanks.
Answers.UserDisplayName: puradrogasincortar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T19:35:14.927Z
Answers.Text: I think thats the right approach indeed. If the answer sufficiently answers your question make sire to upvote it and select it as the correct answer
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T13:04:30.477Z
Answers.LastActivityDate: 2023-04-12T13:04:30.477Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I have a pytorch model (custom model inherited from nn.Module).</p>
  <p>I'm developing some architecture, for which makes sense for my task to have a list defined in the model as:</p>
  <pre><code>class Model(nn.Module):
     def __init__(self, hparams):
         super().__init__()
         self.list = []
  </code></pre>
  <p>In every training step, I want to store in this list some information. Particularly, a number that I will calculate over the training step.</p>
  <p>To be more clear:</p>
  <p>for each training step I receive a batch of size <span class="math-container">$n$</span>, and in the training step I update the <code>self.list</code> in <code>Model</code> to store <span class="math-container">$n$</span> values. For instance, after a first iteration with a batch size of 100, the <code>self.list</code> list will contain 100 values. Every iteration, I will add 100 more to the list...</p>
  <p>I mean, this makes sense for my task, but I'm wondering if this is &quot;legal&quot;. Also it seems a bit weird, because the parameters of my model would increase with the training size? Because in the end, if I use 10,000 training samples, this list will hold 10,000 values. But if I use more, the list will have more. Would this be an increase in parameters? When I save this model, will the number of parameters be affected by this? That's what I'm not sure about.</p>
  
Tags: <neural-networks><pytorch><weights>
Title: Does it make sense to store information in a variable defined inside a Pytorch nn.Module?
UserDisplayName: puradrogasincortar
ThreadText: # Does it make sense to store information in a variable defined inside a Pytorch nn.Module?
  
  (Asked by: username_0 on 2023-04-12)
  
  I have a pytorch model (custom model inherited from nn.Module).
  
  I'm developing some architecture, for which makes sense for my task to have a list defined in the model as:
  
  ```
  class Model(nn.Module):
     def __init__(self, hparams):
         super().__init__()
         self.list = []
  
  ```
  
  In every training step, I want to store in this list some information. Particularly, a number that I will calculate over the training step.
  
  To be more clear:
  
  for each training step I receive a batch of size $n$, and in the training step I update the `self.list` in `Model` to store $n$ values. For instance, after a first iteration with a batch size of 100, the `self.list` list will contain 100 values. Every iteration, I will add 100 more to the list...
  
  I mean, this makes sense for my task, but I'm wondering if this is "legal". Also it seems a bit weird, because the parameters of my model would increase with the training size? Because in the end, if I use 10,000 training samples, this list will hold 10,000 values. But if I use more, the list will have more. Would this be an increase in parameters? When I save this model, will the number of parameters be affected by this? That's what I'm not sure about.
  
  **Tags:** neural-networks, pytorch, weights
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-12. Score: 1 | Accepted answer
  
  Generally, your PyTorch module is to contain the architecture that you are training, not the additional variables that you want to store. So generally, I would not implement my list like this.
  
  What is usually done is to have one `nn.Module` for the architecture and one generic class for utility functions such as the model training/fitting/validation etc. Then the module is passed to the generic 'Trainer' class for training the model and your list of values would presumably be stored in the 'Trainer' object.
  
  So something like this:
  
  ```
  class MyNetwork(nn.Module):
      # Here define the architecture
      def __init__(self, *other_arguments): ...
     
      def forward(batch): ...
  
  class Trainer():
      def __init__(self, network_instance, optimizer, *other_arguments):
          self.network_instance = network_instance
          self.optimizer = optimizer
  
          self.list_of_values = []
  
      def train(n_epochs):
          for i in range(n_epochs):
              # Standard train function layout
              self.optimizer.zero_grad()
              batch = self.get_data()
              predictions = self.network_instance(batch)
              loss = calculate_loss(predictions)
              loss.backward()
              self.optimizer.step()
  
              # Add value to list
              self.list_of_values.append(loss.item())
  
  ```
  
  > **Comments:**
  > **username_0**: I understand. It indeed makes sense to store the list in the Trainer. Now, if I store a separate model (let's say an sklearn model) in a variable of the nn.Module, and I use this accumulated list to train that model, (calling it in the trainer as `sklearn_model.fit(list_of_values)`, will those learned weights be succesfully stored inside the model variable in the nn.Module after training?  
  > **username_1**: I am not familiar with the integration of sklearn models (and their parameters) into PyTorch Modules. I'd advise keeping them separate if possible.  
  > **username_0**: I actually think they are not made to work together in terms of saving sklearn model weights for instance, at least from my experiments. I guess the right choice would be to implement the sklearn model in Pytorch and then load this model as an instance of the nn.Module class. Many thanks.  
  > **username_1**: I think thats the right approach indeed. If the answer sufficiently answers your question make sire to upvote it and select it as the correct answer  
  
  ---
  
  