Answers.Body: <p>You could use <a href="https://en.wikipedia.org/wiki/Mutual_information" rel="nofollow noreferrer">Mutual Information</a> between the model's prediction, and that particular feature as a regularization term. This will minimize the dependence of the output to that particular feature. Note that simply removing the feature from the dataset might not work if other features are associated with the feature which you don't want your model to depend on.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-05T14:56:58.423Z
Answers.Text: That sounds reasonable. After searching through Mutual information loss, I didn't find easy to read references. Can you please point me to some of the references that explain/implement mutual information loss?
Answers.UserDisplayName: pg2455
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-07T09:27:31.573Z
Answers.Text: @pg2455 I'm afraid I don't have reference material regarding implementation. You might wanna try PyTorch forums in case you're using that framework.
Answers.UserDisplayName: SpiderRico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-29T19:45:56.893Z
Answers.LastActivityDate: 2021-09-29T19:55:39.793Z
Answers.LastEditDate: 2021-09-29T19:55:39.793Z
Answers.UserDisplayName: SpiderRico
Answers.Body: <pre><code>    def call(self, x):
          x_no_x6 = tf.concat([x[:,:5], x[:,5+1:]], axis=1)
          f2 = ... model architecture goes here ... (function of x_no_x6)
          f1 = ... model architecture... (function of x)
          return tf.concat([f1, f2], axis=1)
  </code></pre>
  <p>You could have two models, one of which uses <span class="math-container">$x$</span> (including <span class="math-container">$x_6$</span>) and the other which removes <span class="math-container">$x_6$</span> from the input. The overall model is just the concatenation of the two outputs. (Pseudo-code is assuming model is implemented by subclassing tensorflow.keras.Model <a href="https://www.tensorflow.org/guide/keras/custom_layers_and_models#the_model_class" rel="nofollow noreferrer">link</a>)</p>
  <p>Not sure about the gradient penalization idea. Seems nontrivial to implement and it's not clear to me if it would work.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T16:02:14.240Z
Answers.Text: Not sure if it is doing what I am looking for. Your output is still dependent on x_6 through f_1. In my question, I am limited to just one function $f$. In your solution, $f_1$ and $f_2$ are different functions with different parameters.
Answers.UserDisplayName: pg2455
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T18:53:37.587Z
Answers.Text: No. There is a single function $f \in \mathcal{R}^2$, and the second component is independent of $x_6$. I think you might be confused because you are not familiar with model subclassing in keras. When training/doing inference etc. you just have the single function (model.call) which goes from $x$ to R2. The parameters are model.trainable_variables, a single set of parameters.
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-01T11:40:21.777Z
Answers.Text: Yeah. I wasn't familiar with subclassing as I mainly use PyTorch. However, I am limited to just one forward computation of $f$ because of time constraints. The suggested solution seems to be doing forward passes, is that correct?
Answers.UserDisplayName: pg2455
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T19:02:27.190Z
Answers.LastActivityDate: 2021-08-30T19:02:27.190Z
Answers.UserDisplayName: Taw
Body: <p>Is there a way to make a certain output dimension of a neural network independent of a particular feature dimension? For example, I have a function <span class="math-container">$f_{\theta} : \mathcal{R}^{10} \rightarrow \mathcal{R}^2$</span>, I want to make <span class="math-container">$f_{\theta}(\mathbf{x})_2$</span>  independent of <span class="math-container">$\mathbf{x}_6$</span>. How can this condition be imposed on a neural network?</p>
  <p>I am thinking of penalizing the gradient of <span class="math-container">$f_{\theta}(\mathbf{x})$</span> w.r.t <span class="math-container">$\mathbf{x}_6$</span> for a considerable range of <span class="math-container">$\mathbf{x}_6 \in [-1, 1]$</span>. Will this give me the similar effect? If so, how can this be coded in Pytorch or any other deep learning framework?</p>
  
Tags: <neural-networks><deep-learning>
Title: How to make an output independent of input feature in neural networks?
UserDisplayName: pg2455
ThreadText: # How to make an output independent of input feature in neural networks?
  
  (Asked by: username_0 on 2021-08-30)
  
  Is there a way to make a certain output dimension of a neural network independent of a particular feature dimension? For example, I have a function $f\_{\theta} : \mathcal{R}^{10} \rightarrow \mathcal{R}^2$, I want to make $f\_{\theta}(\mathbf{x})\_2$ independent of $\mathbf{x}\_6$. How can this condition be imposed on a neural network?
  
  I am thinking of penalizing the gradient of $f\_{\theta}(\mathbf{x})$ w.r.t $\mathbf{x}\_6$ for a considerable range of $\mathbf{x}\_6 \in [-1, 1]$. Will this give me the similar effect? If so, how can this be coded in Pytorch or any other deep learning framework?
  
  **Tags:** neural-networks, deep-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-29. Score: 1
  
  You could use [Mutual Information](https://en.wikipedia.org/wiki/Mutual_information) between the model's prediction, and that particular feature as a regularization term. This will minimize the dependence of the output to that particular feature. Note that simply removing the feature from the dataset might not work if other features are associated with the feature which you don't want your model to depend on.
  
  > **Comments:**
  > **username_0**: That sounds reasonable. After searching through Mutual information loss, I didn't find easy to read references. Can you please point me to some of the references that explain/implement mutual information loss?  
  > **username_1**: @pg2455 I'm afraid I don't have reference material regarding implementation. You might wanna try PyTorch forums in case you're using that framework.  
  
  ---
  
  ### Answer by username_2 on 2021-08-30. Score: 0
  
  def call(self, x):
          x_no_x6 = tf.concat([x[:,:5], x[:,5+1:]], axis=1)
          f2 = ... model architecture goes here ... (function of x_no_x6)
          f1 = ... model architecture... (function of x)
          return tf.concat([f1, f2], axis=1)
  
  You could have two models, one of which uses $x$ (including $x_6$) and the other which removes $x_6$ from the input. The overall model is just the concatenation of the two outputs. (Pseudo-code is assuming model is implemented by subclassing tensorflow.keras.Model link)
  Not sure about the gradient penalization idea. Seems nontrivial to implement and it's not clear to me if it would work.
  
  > **Comments:**
  > **username_0**: Not sure if it is doing what I am looking for. Your output is still dependent on x_6 through f_1. In my question, I am limited to just one function $f$. In your solution, $f_1$ and $f_2$ are different functions with different parameters.  
  > **username_2**: No. There is a single function $f \in \mathcal{R}^2$, and the second component is independent of $x_6$. I think you might be confused because you are not familiar with model subclassing in keras. When training/doing inference etc. you just have the single function (model.call) which goes from $x$ to R2. The parameters are model.trainable_variables, a single set of parameters.  
  > **username_0**: Yeah. I wasn't familiar with subclassing as I mainly use PyTorch. However, I am limited to just one forward computation of $f$ because of time constraints. The suggested solution seems to be doing forward passes, is that correct?  
  
  ---
  
  