Answers.Body: <p>Yes, you can define a totally custom Model, maybe with a clustering method you can call after <code>forward</code> only during inference.
  Clustering parameters (e.g centroids in Kmeans) can be stored inside a buffer (see <a href="https://stackoverflow.com/questions/59620431/what-is-a-buffer-in-pytorch">here</a> and <a href="https://stackoverflow.com/questions/57540745/what-is-the-difference-between-register-parameter-and-register-buffer-in-pytorch/57546078#57546078">here</a>)</p>
  <p>Clustering is not typically optimized with Gradient Descent, so clustering parameters should not require gradient, and you need to pay attention to not include the clustering step into your computational graph.</p>
  <p>I would implement this pipeline:</p>
  <ol>
  <li><strong>Autoencoder training</strong>: Train your AutoEncoder as usual, and save the model's checkpoint.</li>
  <li><strong>Feature extraction</strong>: Extract all latent codes using your trained autoencoder, then save them onto disk into a numpy array or a tensor.</li>
  <li><strong>Clustering</strong>: Load all latent codes and apply a clustering algorithm (e.g Kmeans) and save all relevant parameters (centroids) onto a buffer .</li>
  <li><strong>Inference</strong>: Load trained autoencoder and centroids. Both parameters and buffers are into model's <code>state_dict</code>, so they can be loaded with the <code>load_state_dict</code> method. Pass new data into the encoder, then cluster the latent representation onto the closest centroid</li>
  </ol>
  <p>By using a modular approach, you can experiment different clustering methods without losing all the features (saving additional training), and then put all together once you're satisfied with the model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T17:27:48.110Z
Answers.LastActivityDate: 2023-04-14T17:27:48.110Z
Answers.UserDisplayName: Ciodar
Body: <p>I am making a custom Pytorch model that at some point, clusters a latent space that was created by another, previous routine of the model (Autoencoder).</p>
  <p>In a bit more detail, my model is a regular Autoencoder, but in every training step, I want to perform a clustering of the representations in the latent space <code>Z</code>, and save the clustering algorithm inside the model to use after training, ONLY for inference (to make certain kind of predictions).</p>
  <p>I thought about defining an sklearn model as an instance of the nn.Module as</p>
  <pre><code>class Model(Module):
      def __init__(self, hparams):
          super().__init__()
          self.clustering_model = sklearn.clustering_model()
  
  </code></pre>
  <p>This way, I could call the clustering model to fit the latent space Z, after calculating the reconstruction loss as a regular autoencoder.</p>
  <p>After training, I would then have in the variable <code>model.clustering_model</code> a trained clustering model that I could use for inference.</p>
  <p>However, this doesn't seem to work out, probably because of incompatibilities between sklearn and Pytorch. Basically, it seems that training the <code>clustering_model</code> and saving it inside the pytorch model didn't actually save any trained weights.</p>
  <p>I'm right now unsure whether it didn't save weights because it actually wasn't trained (by an error on my side) or if it is that PyTorch just can't save a sklearn trained model as a variable in a <code>nn.Module</code>.</p>
  <p>Did anyone try before to implement sklearn inside a nn.Module?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-13T15:44:18.110Z
Comments.Text: I've not used Pytorch before, but [skorch](https://github.com/skorch-dev/skorch) might be a helpful library for you.
Comments.UserDisplayName: LittleLulatsch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-13T21:27:51.387Z
Comments.Text: @LittleLulatsch I've checked! Sadly the library is made to transport Pytorch to sklearn, and not the other way around.
Comments.UserDisplayName: puradrogasincortar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T06:31:58.383Z
Comments.Text: It is indeed possible, yet It may be trickier than just add the clustering code to the forward method of autoencoder. Are there any particular reasons to not do it? forward method also allows to return several values.
Comments.UserDisplayName: Ciodar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T08:03:02.017Z
Comments.Text: @Ciodar Do you mean that they should be saved if they are added to the forward method instead of just the class? I'm unsure about which is the criteria of Pytorch to save information about a model instance. It seems that Pytorch will only save weights that were trained (i.e. that were part of the loss), but my clustering algorithm wouldn't enter into the pytorch loss calculation.
Comments.UserDisplayName: puradrogasincortar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T08:17:11.657Z
Comments.Text: Maybe I misunderstood the first time I read that. Do you want to compute a latent representation $z_k$ for each of your input samples, and then cluster all latent representations into a latent subspace? In this case, I think the simpler solution is to split these two problems, storing all output latent spaces and performing a clustering of those.
Comments.UserDisplayName: Ciodar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T08:23:30.487Z
Comments.Text: Once you have computed the centroids, you can save your clustering params and restore them. I'm kinda wary to actually introduce Kmeans parameters as module parameters, since they should not be added into the computational graph (Kmeans is not typically trained with GD)
Comments.UserDisplayName: Ciodar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T08:36:50.820Z
Comments.Text: One solution to have them in your custom Pytorch model is to register a [buffer](https://stackoverflow.com/questions/59620431/what-is-a-buffer-in-pytorch) where you store your clustering parameters, and define a procedure to save/restore the clustering weights from the buffer. Then you can call the clustering algorithm (maybe only during prediction) using a custom method or also in your forward. (I personally would call a custom method)
Comments.UserDisplayName: Ciodar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T11:17:10.067Z
Comments.Text: @Ciodar Indeed a nn.parameters is what I need to tell Pytorch to save those weights. Thanks!
Comments.UserDisplayName: puradrogasincortar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-14T12:02:46.810Z
Comments.Text: I would use buffers instead, as specified in the link above and also [here](https://stackoverflow.com/questions/57540745/what-is-the-difference-between-register-parameter-and-register-buffer-in-pytorch/57546078#57546078). Anyways, I will transform these comments in an answer if they fit your needs.
Comments.UserDisplayName: Ciodar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T12:03:17.797Z
Comments.Text: Programming questions (which includes questions related to specific libraries) are off-topic here. Please, ask them at Stack Overflow or Data Science SE. See our on-topic page. We focus on the theoretical aspects of AI.
Comments.UserDisplayName: nbro
Tags: <pytorch><autoencoders><unsupervised-learning><scikit-learn>
Title: Can I implement a sklearn model inside a Pytorch nn.Module?
UserDisplayName: puradrogasincortar
ThreadText: # Can I implement a sklearn model inside a Pytorch nn.Module?
  
  (Asked by: username_0 on 2023-04-13)
  
  I am making a custom Pytorch model that at some point, clusters a latent space that was created by another, previous routine of the model (Autoencoder).
  
  In a bit more detail, my model is a regular Autoencoder, but in every training step, I want to perform a clustering of the representations in the latent space `Z`, and save the clustering algorithm inside the model to use after training, ONLY for inference (to make certain kind of predictions).
  
  I thought about defining an sklearn model as an instance of the nn.Module as
  
  ```
  class Model(Module):
      def __init__(self, hparams):
          super().__init__()
          self.clustering_model = sklearn.clustering_model()
  
  ```
  
  This way, I could call the clustering model to fit the latent space Z, after calculating the reconstruction loss as a regular autoencoder.
  
  After training, I would then have in the variable `model.clustering_model` a trained clustering model that I could use for inference.
  
  However, this doesn't seem to work out, probably because of incompatibilities between sklearn and Pytorch. Basically, it seems that training the `clustering_model` and saving it inside the pytorch model didn't actually save any trained weights.
  
  I'm right now unsure whether it didn't save weights because it actually wasn't trained (by an error on my side) or if it is that PyTorch just can't save a sklearn trained model as a variable in a `nn.Module`.
  
  Did anyone try before to implement sklearn inside a nn.Module?
  
  **Tags:** pytorch, autoencoders, unsupervised-learning, scikit-learn
  
  > **Comments:**
  > **username_1**: I've not used Pytorch before, but [skorch](https://github.com/skorch-dev/skorch) might be a helpful library for you.  
  > **username_0**: @username_1 I've checked! Sadly the library is made to transport Pytorch to sklearn, and not the other way around.  
  > **username_2**: It is indeed possible, yet It may be trickier than just add the clustering code to the forward method of autoencoder. Are there any particular reasons to not do it? forward method also allows to return several values.  
  > **username_0**: @username_2 Do you mean that they should be saved if they are added to the forward method instead of just the class? I'm unsure about which is the criteria of Pytorch to save information about a model instance. It seems that Pytorch will only save weights that were trained (i.e. that were part of the loss), but my clustering algorithm wouldn't enter into the pytorch loss calculation.  
  > **username_2**: Maybe I misunderstood the first time I read that. Do you want to compute a latent representation $z_k$ for each of your input samples, and then cluster all latent representations into a latent subspace? In this case, I think the simpler solution is to split these two problems, storing all output latent spaces and performing a clustering of those.  
  > **username_2**: Once you have computed the centroids, you can save your clustering params and restore them. I'm kinda wary to actually introduce Kmeans parameters as module parameters, since they should not be added into the computational graph (Kmeans is not typically trained with GD)  
  > **username_2**: One solution to have them in your custom Pytorch model is to register a [buffer](https://stackoverflow.com/questions/59620431/what-is-a-buffer-in-pytorch) where you store your clustering parameters, and define a procedure to save/restore the clustering weights from the buffer. Then you can call the clustering algorithm (maybe only during prediction) using a custom method or also in your forward. (I personally would call a custom method)  
  > **username_0**: @username_2 Indeed a nn.parameters is what I need to tell Pytorch to save those weights. Thanks!  
  > **username_2**: I would use buffers instead, as specified in the link above and also [here](https://stackoverflow.com/questions/57540745/what-is-the-difference-between-register-parameter-and-register-buffer-in-pytorch/57546078#57546078). Anyways, I will transform these comments in an answer if they fit your needs.  
  > **username_3**: Programming questions (which includes questions related to specific libraries) are off-topic here. Please, ask them at Stack Overflow or Data Science SE. See our on-topic page. We focus on the theoretical aspects of AI.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-14. Score: 1 | Accepted answer
  
  Yes, you can define a totally custom Model, maybe with a clustering method you can call after `forward` only during inference.
  Clustering parameters (e.g centroids in Kmeans) can be stored inside a buffer (see [here](https://stackoverflow.com/questions/59620431/what-is-a-buffer-in-pytorch) and [here](https://stackoverflow.com/questions/57540745/what-is-the-difference-between-register-parameter-and-register-buffer-in-pytorch/57546078#57546078))
  
  Clustering is not typically optimized with Gradient Descent, so clustering parameters should not require gradient, and you need to pay attention to not include the clustering step into your computational graph.
  
  I would implement this pipeline:
  
  1. **Autoencoder training**: Train your AutoEncoder as usual, and save the model's checkpoint.
  2. **Feature extraction**: Extract all latent codes using your trained autoencoder, then save them onto disk into a numpy array or a tensor.
  3. **Clustering**: Load all latent codes and apply a clustering algorithm (e.g Kmeans) and save all relevant parameters (centroids) onto a buffer .
  4. **Inference**: Load trained autoencoder and centroids. Both parameters and buffers are into model's `state_dict`, so they can be loaded with the `load_state_dict` method. Pass new data into the encoder, then cluster the latent representation onto the closest centroid
  
  By using a modular approach, you can experiment different clustering methods without losing all the features (saving additional training), and then put all together once you're satisfied with the model.
  
  
  ---
  
  