Answers.Body: <p>This is kind of a black-box optimization problem, but luckily you can calculate gradients between its input and output. Since the number of inputs is so small, you could just brute-force it by evaluating the network for a <a href="https://en.wikipedia.org/wiki/Hyperparameter_optimization#Grid_search" rel="nofollow noreferrer">grid</a> of input samples. Naturally you'll need to choose how many samples to do along each dimension. Since the network has quite many parameters, I expect the &quot;surface&quot; to be quite intricate.</p>
  <p>A more efficient approach is to run a gradient descent algorithm on it, starting from various random points. Then again the brute-force approach will take better advantage of a GPU.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-21T16:56:01.333Z
Answers.LastActivityDate: 2022-11-21T16:56:01.333Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I have a feed forward neural network defined in pytorch as follows:</p>
  <pre><code>    model = torch.nn.Sequential(
          torch.nn.Linear(2, 50),
          torch.nn.GELU(approximate='tanh'),
          torch.nn.Linear(50, 50),
          torch.nn.GELU(approximate='tanh'),
          torch.nn.Linear(50, 1),
      )
  </code></pre>
  <p>The model has been trained.  Now, I have specific intervals of input values (e.g. [0, 0.1), [0.1, 02.),...[0.9, 1]) within which I want to find the maximum and minimum values of the model's output.  I am not looking for specific code or software to do this.  Rather, what are some general strategies for doing this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-21T13:40:43.530Z
Comments.Text: this is a question more suited for StackOverflow since it's about programming. Anyway, you can check the pytorch documentation to find the answer to these type of questions, start from https://pytorch.org/docs/stable/generated/torch.argmax.html
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-21T13:53:21.013Z
Comments.Text: @EdoardoGuerriero I added the pytorch model definition for context, but am not expecting a code answer. I'm looking for a theoretical answer on how the solution could be computed. I'm happy to figure out how to do the implementation myself.
Comments.UserDisplayName: peocje
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-21T14:26:35.080Z
Comments.Text: ok I see, maybe the phrasing is a bit unclear, are you asking if it's possible to know beforehand, i.e. without running a model,  the range of a network predictions based on the range of its inputs?
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-21T14:37:37.030Z
Comments.Text: @EdoardoGuerriero Apologies if it was unclear. I have the trained model, and am willing to run it. But it's unclear to me how you would run it when the input would be intervals rather than fixed values. I.E rather than doing inference on input (0.3, 0.4), I want to do inference on ([0.3, 0.35], [0.4, 0.49]).
Comments.UserDisplayName: peocje
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-21T20:52:12.553Z
Comments.Text: You can only input into your model the type of data that you used to train it.  Your model, therefore, **will not accept intervals.**  As suggest in the answer by @NikoNyrh, you can approximate the minimum and maximum values in the interval by sampling within your interval.  For example, the $[0, 0.1]$ interval can be sampled as $0, 0.01, 0.02, ...0.09, 0.1$.  Then, run each number in the sampling **individually** as an input into the model (perhaps in a `for` loop) and collect outputs in a list.  Find the min and max of the list.  Repeat for each interval.
Comments.UserDisplayName: Snehal Patel
Tags: <neural-networks><feedforward-neural-networks>
Title: Find maxima and minima of feed forward neural network given interval inputs
UserDisplayName: peocje
ThreadText: # Find maxima and minima of feed forward neural network given interval inputs
  
  (Asked by: username_0 on 2022-11-21)
  
  I have a feed forward neural network defined in pytorch as follows:
      model = torch.nn.Sequential(
          torch.nn.Linear(2, 50),
          torch.nn.GELU(approximate='tanh'),
          torch.nn.Linear(50, 50),
          torch.nn.GELU(approximate='tanh'),
          torch.nn.Linear(50, 1),
      )
  
  The model has been trained.  Now, I have specific intervals of input values (e.g. [0, 0.1), [0.1, 02.),...[0.9, 1]) within which I want to find the maximum and minimum values of the model's output.  I am not looking for specific code or software to do this.  Rather, what are some general strategies for doing this?
  
  **Tags:** neural-networks, feedforward-neural-networks
  > **Comments:**
  > **username_1**: this is a question more suited for StackOverflow since it's about programming. Anyway, you can check the pytorch documentation to find the answer to these type of questions, start from https://pytorch.org/docs/stable/generated/torch.argmax.html  
  > **username_0**: @EdoardoGuerriero I added the pytorch model definition for context, but am not expecting a code answer. I'm looking for a theoretical answer on how the solution could be computed. I'm happy to figure out how to do the implementation myself.  
  > **username_1**: ok I see, maybe the phrasing is a bit unclear, are you asking if it's possible to know beforehand, i.e. without running a model,  the range of a network predictions based on the range of its inputs?  
  > **username_0**: @EdoardoGuerriero Apologies if it was unclear. I have the trained model, and am willing to run it. But it's unclear to me how you would run it when the input would be intervals rather than fixed values. I.E rather than doing inference on input (0.3, 0.4), I want to do inference on ([0.3, 0.35], [0.4, 0.49]).  
  > **username_2**: You can only input into your model the type of data that you used to train it.  Your model, therefore, **will not accept intervals.**  As suggest in the answer by @NikoNyrh, you can approximate the minimum and maximum values in the interval by sampling within your interval.  For example, the $[0, 0.1]$ interval can be sampled as $0, 0.01, 0.02, ...0.09, 0.1$.  Then, run each number in the sampling **individually** as an input into the model (perhaps in a `for` loop) and collect outputs in a list.  Find the min and max of the list.  Repeat for each interval.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-11-21. Score: 0 | Accepted answer
  
  This is kind of a black-box optimization problem, but luckily you can calculate gradients between its input and output. Since the number of inputs is so small, you could just brute-force it by evaluating the network for a [grid](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Grid_search) of input samples. Naturally you'll need to choose how many samples to do along each dimension. Since the network has quite many parameters, I expect the "surface" to be quite intricate.
  
  A more efficient approach is to run a gradient descent algorithm on it, starting from various random points. Then again the brute-force approach will take better advantage of a GPU.
  
  
  ---
  
  