Answers.Body: <p>An interesting problem. This network has only 933 trainable parameters, and obtains <code>MeanAbsolutePercentageError</code> of 0.01 - 0.04. It is based on a softmax activation, to choose which item from the input to choose.</p>
  <pre><code>n, dim, validation_split = 100000, 5, 0.1
  X = tf.random.uniform(shape=(n, dim), minval=0.0, maxval=1.0)
  y = tf.reduce_max(X, axis=-1)
  
  initial_learning_rate = 0.003
  act, units, depth = 'relu', 16, 3
  
  inp = tf.keras.layers.Input(dim)
  x = inp
  
  for _ in range(depth):
      x = tf.keras.layers.Dense(units, activation=act)(x)
      x = tf.keras.layers.BatchNormalization()(x)
  
  x = tf.keras.layers.Dense(int((dim * units)**0.5), activation=&quot;elu&quot;)(x)
  x = tf.keras.layers.BatchNormalization()(x)
  
  x = tf.keras.layers.Dense(dim, activation=&quot;softmax&quot;)(x)
  
  # Skip this step to make categorical predictions
  x = tf.reduce_sum(inp * x, axis=-1, keepdims=True)
  
  model = tf.keras.models.Model(inp, x)
  model.summary()
  
  is_categorical = model.output.shape[1] == dim
  
  model.compile(
      optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
      loss=tf.keras.losses.CategoricalCrossentropy() if is_categorical else
           tf.keras.losses.MeanAbsolutePercentageError(),
      metrics=[tf.keras.metrics.CategoricalAccuracy()] if is_categorical else []
  )
  
  stopping_callback = tf.keras.callbacks.EarlyStopping(monitor=&quot;val_loss&quot;, patience=10, restore_best_weights=True)
  reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor=&quot;loss&quot;, factor=0.1**0.5, patience=5,
                                                            verbose=1, min_lr=1e-4)
  
  target = tf.cast(X == y[:,None], tf.float32) if is_categorical else y
  
  h = model.fit(X, target, verbose=1,
                batch_size=1024, epochs=10000, validation_split=validation_split,
                callbacks=[stopping_callback, reduce_lr_callback])
  
  n_val = int(n * validation_split)
  print([model.evaluate(X[:n_val],  target[:n_val]),
         model.evaluate(X[-n_val:], target[-n_val:])])
  </code></pre>
  <p>An alternative formulation outputs just the softmax activation, this obtains an accuracy of 98 - 99%. Granted, it isn't 100% accurate.</p>
  <p>Summary of the model with just one output:</p>
  <pre><code>__________________________________________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_273 (InputLayer)          [(None, 5)]          0                                            
  __________________________________________________________________________________________________
  dense_868 (Dense)               (None, 16)           96          input_273[0][0]                  
  __________________________________________________________________________________________________
  batch_normalization_577 (BatchN (None, 16)           64          dense_868[0][0]                  
  __________________________________________________________________________________________________
  dense_869 (Dense)               (None, 16)           272         batch_normalization_577[0][0]    
  __________________________________________________________________________________________________
  batch_normalization_578 (BatchN (None, 16)           64          dense_869[0][0]                  
  __________________________________________________________________________________________________
  dense_870 (Dense)               (None, 16)           272         batch_normalization_578[0][0]    
  __________________________________________________________________________________________________
  batch_normalization_579 (BatchN (None, 16)           64          dense_870[0][0]                  
  __________________________________________________________________________________________________
  dense_871 (Dense)               (None, 8)            136         batch_normalization_579[0][0]    
  __________________________________________________________________________________________________
  batch_normalization_580 (BatchN (None, 8)            32          dense_871[0][0]                  
  __________________________________________________________________________________________________
  dense_872 (Dense)               (None, 5)            45          batch_normalization_580[0][0]    
  __________________________________________________________________________________________________
  tf_op_layer_mul_251 (TensorFlow [(None, 5)]          0           input_273[0][0]                  
                                                                   dense_872[0][0]                  
  __________________________________________________________________________________________________
  tf_op_layer_Sum_329 (TensorFlow [(None, 1)]          0           tf_op_layer_mul_251[0][0]        
  ==================================================================================================
  Total params: 1,045
  Trainable params: 933
  Non-trainable params: 112
  __________________________________________________________________________________________________
  </code></pre>
  <p>I don't know what kind your original problem is, but maybe you could do some feature-engineering by supplying the min/max/median directly to the network. Then the network wouldn't need to learn to approximate those functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T22:31:40.713Z
Answers.Text: Thank you! The secret here seems to be this layer:
  x = tf.reduce_sum(inp * x, axis=-1, keepdims=True)
  Can you give any intuition or guidance as to why you decided to do it this way?
  The elu activation layer with a very specific number of units is also interesting, can you shed some light on that? And the idea of putting in the min/max/median, I don't think I've seen that anywhere before?
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T23:38:22.073Z
Answers.Text: I also noticed this doesn't seem to extrapolate well to other functions - e.g. if we use reduce_min() instead of reduce_max(), it doesn't work at all
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T00:31:33.990Z
Answers.Text: For reference, one of the most simple functions I actually care about approximating is something like this: 
  
  https://gist.github.com/JustASquid/376ecc8f6469fb5a4262941c293352ab
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T20:57:32.440Z
Answers.Text: The `tf.reduce_sum(inp * x, axis=-1, keepdims=True)` came from a hunch that maybe the `softmax` would learn to pick the correct input, and push the weight of the others to zero. It is surprising that this didn't work when the aggregation is `min`! However the categorical prediction works on both cases, but clearly you'd like to solve this as a regression problem instead. I'll post an update if something comes to my mind.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T20:59:20.367Z
Answers.Text: The number of dimensions in `elu` is the geometric mean of its input and output size, I have a habbit of doing that. And I have found that `elu` works better in regression tasks than `relu`, actually it can approximate `relu` (or a plain linear function) arbitrary well when combined with batch normalization and subsequent layers.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T21:02:50.680Z
Answers.Text: Actually in addition to just min/max/median, if you input is large (like tens or hundreds of samples) then you should try augmenting the input with 0th, 10th, 20th, ...50th, ... 90th and 100th percentiles of each row. It is nice that these statistics are invariant to the order of input samples, and hopefully the network will learn to use them to predict the correct output.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T01:53:04.680Z
Answers.Text: it's actually potentially feasible for my application to do it with a categorical selection. However, your solution with an accuracy of 98-99% is also not acceptable, I need an accuracy of 99.99%+ before I could consider this to be feasible.
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T21:33:53.260Z
Answers.Text: I think the classification task is unfeasible to solve accurately and reliably with a neural network, since the correct output is very sensitive to the inputs. Like having inputs `0, 0, 0.99, 1.01, 0` has a different correct answer than `0, 0, 1.01, 0.99, 0` (this can be generalized to any epsilon).
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T21:39:43.970Z
Answers.Text: That's a good perspective. I would have to imagine it's possible to design some architecture/layer that deals with these cases better? I wonder if there's anything in the literature about this?
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T21:53:29.750Z
Answers.Text: I think it is very difficult, wihtout having explicit min/max/percentile inputs to the network. And even then, a regression network might learn that the correct output is `max(input) + 1e-3 * x[0] - 1e-5 * x[1]` etc. which is always incorrect by some amount.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T22:37:50.437Z
Answers.Text: maybe something closer to a decision tree would work better? If I can find something differentiable then it could still work for my application...
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T18:12:15.050Z
Answers.LastActivityDate: 2023-01-09T18:12:15.050Z
Answers.UserDisplayName: NikoNyrh
Answers.Body: <p>Assuming the <span class="math-container">$N$</span> numbers are different, a function that combines sums, products and the hard step function for solving the maximum may look like this.</p>
  <p><span class="math-container">$$ max(x_1,x_2,...,x_N) = \sum_{i=1}^N x_i \prod_{j=1,j\neq i}^N sign(x_i-x_j) $$</span></p>
  <p><span class="math-container">$sign(z)$</span> is <span class="math-container">$1$</span> for <span class="math-container">$z&gt;0$</span> and <span class="math-container">$0$</span> for <span class="math-container">$z&lt;0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T22:34:20.620Z
Answers.Text: The example is just to demonstrate regressing some unknown function of N inputs with 1 output. You can imagine replacing max() with something else (e.g. sum() ) and the problem is still valid.
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T12:32:06.643Z
Answers.LastActivityDate: 2023-01-09T12:32:06.643Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm facing.</p>
  <p>One could attempt to train an MLP to solve this problem using randomly generated input data. The following code is an example in Keras/Tensorflow. It generates 1000000 random examples (A large number, so we can reduce overfitting as a factor). It also applies some simple techniques like LR decay and early stopping to optimize training.</p>
  <pre><code>import tensorflow as tf
  
  # Create a dataset where the y value is the largest of 5 random x values.
  x = tf.random.uniform(shape=(1000000, 5), minval=0.0, maxval=1.0)
  y = tf.reduce_max(x, axis=-1)
  
  dataset = tf.data.Dataset.from_tensor_slices(
      (x, y)
  ).shuffle(1000000).batch(1024).prefetch(1)
  
  # See the post for discussion of hyperparameter optimization
  units = 400
  depth = 5
  initial_learning_rate = 0.001
  
  model = tf.keras.models.Sequential()
  
  for i in range(depth):
      model.add(tf.keras.layers.Dense(units, activation=&quot;relu&quot;))
  model.add(tf.keras.layers.Dense(1))
  
  model.compile(
      optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
      loss=tf.keras.losses.MeanAbsolutePercentageError(),
  )
  
  stopping_callback = tf.keras.callbacks.EarlyStopping(monitor=&quot;loss&quot;, patience=5)
  
  reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor=&quot;loss&quot;, factor=0.75, patience=2)
  
  model.fit(dataset, epochs=1000)
  </code></pre>
  <p>The metric we care about is the mean absolute percentage error. When we run the code, the model trains successfully. But very importantly, the loss does not ever go below ~0.2 with these hyperparameters.</p>
  <p>0.2% error sounds pretty good, but it's actually unacceptable for my use case. I need much more precision, as in practice the model is used in a situation where any errors are magnified.</p>
  <p>At this point, you may want to suggest hyperparameter tuning, but I assure you I have done an extensive amount of hyperparameter tuning on this and similar problems.</p>
  <p>The model is clearly underfitting, and it is possible to decrease the error by increasing the number of parameters in the network. Empirically, I've noticed that increasing the number of parameters (and number of examples) by an order of magnitude can reduce the error by a similar order of magnitude.</p>
  <p>However, intuitively this just feels wrong. It is not practical to have a network with millions, or even tens of millions of parameters, just to regress such a simple function. It feels like there should be an architecture that can learn a simple function much more efficiently, but I have not been able to find any reference to such an alternative architecture.</p>
  <p>Again, of course this particular example is trivial, but hopefully you can imagine a real-world analogue where an unknown (more complex, but similar) function is expressed by a similar dataset. And in such a real-world example, the unacceptable error still persists (and is much more pronounced, for reasons that I don't fully understand).</p>
  <p>Does anyone have any idea what's going on here? Am I missing something? How can I improve the performance to a place that would be acceptable?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-09T06:26:54.657Z
Comments.Text: Would softmax (https://en.wikipedia.org/wiki/Softmax_function) help?
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-09T06:28:22.537Z
Comments.Text: @JaumeOliverLafont Not in practice, as far as I can recall. Is there theory behind why softmax would be better than relu here?
Comments.UserDisplayName: Daniel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-09T10:10:57.690Z
Comments.Text: Any reason why you would not model 5 output nodes and have them each score a percentage of how likely it is that that one is the largest?
Comments.UserDisplayName: Robin van Hoorn
Tags: <tensorflow><keras><regression><multilayer-perceptrons>
Title: Why is a simple regression problem so hard for an MLP to learn?
UserDisplayName: Daniel
ThreadText: # Why is a simple regression problem so hard for an MLP to learn?
  
  (Asked by: username_0 on 2023-01-09)
  
  Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm facing.
  
  One could attempt to train an MLP to solve this problem using randomly generated input data. The following code is an example in Keras/Tensorflow. It generates 1000000 random examples (A large number, so we can reduce overfitting as a factor). It also applies some simple techniques like LR decay and early stopping to optimize training.
  
  ```
  import tensorflow as tf
  
  # Create a dataset where the y value is the largest of 5 random x values.
  x = tf.random.uniform(shape=(1000000, 5), minval=0.0, maxval=1.0)
  y = tf.reduce_max(x, axis=-1)
  
  dataset = tf.data.Dataset.from_tensor_slices(
      (x, y)
  ).shuffle(1000000).batch(1024).prefetch(1)
  
  # See the post for discussion of hyperparameter optimization
  units = 400
  depth = 5
  initial_learning_rate = 0.001
  
  model = tf.keras.models.Sequential()
  
  for i in range(depth):
      model.add(tf.keras.layers.Dense(units, activation="relu"))
  model.add(tf.keras.layers.Dense(1))
  
  model.compile(
      optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
      loss=tf.keras.losses.MeanAbsolutePercentageError(),
  )
  
  stopping_callback = tf.keras.callbacks.EarlyStopping(monitor="loss", patience=5)
  
  reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor="loss", factor=0.75, patience=2)
  
  model.fit(dataset, epochs=1000)
  
  ```
  
  The metric we care about is the mean absolute percentage error. When we run the code, the model trains successfully. But very importantly, the loss does not ever go below ~0.2 with these hyperparameters.
  
  0.2% error sounds pretty good, but it's actually unacceptable for my use case. I need much more precision, as in practice the model is used in a situation where any errors are magnified.
  
  At this point, you may want to suggest hyperparameter tuning, but I assure you I have done an extensive amount of hyperparameter tuning on this and similar problems.
  
  The model is clearly underfitting, and it is possible to decrease the error by increasing the number of parameters in the network. Empirically, I've noticed that increasing the number of parameters (and number of examples) by an order of magnitude can reduce the error by a similar order of magnitude.
  
  However, intuitively this just feels wrong. It is not practical to have a network with millions, or even tens of millions of parameters, just to regress such a simple function. It feels like there should be an architecture that can learn a simple function much more efficiently, but I have not been able to find any reference to such an alternative architecture.
  
  Again, of course this particular example is trivial, but hopefully you can imagine a real-world analogue where an unknown (more complex, but similar) function is expressed by a similar dataset. And in such a real-world example, the unacceptable error still persists (and is much more pronounced, for reasons that I don't fully understand).
  
  Does anyone have any idea what's going on here? Am I missing something? How can I improve the performance to a place that would be acceptable?
  
  **Tags:** tensorflow, keras, regression, multilayer-perceptrons
  
  > **Comments:**
  > **username_1**: Would softmax (https://en.wikipedia.org/wiki/Softmax_function) help?  
  > **username_0**: @username_1 Not in practice, as far as I can recall. Is there theory behind why softmax would be better than relu here?  
  > **username_2**: Any reason why you would not model 5 output nodes and have them each score a percentage of how likely it is that that one is the largest?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-01-09. Score: 3
  
  An interesting problem. This network has only 933 trainable parameters, and obtains `MeanAbsolutePercentageError` of 0.01 - 0.04. It is based on a softmax activation, to choose which item from the input to choose.
  
  ```
  n, dim, validation_split = 100000, 5, 0.1
  X = tf.random.uniform(shape=(n, dim), minval=0.0, maxval=1.0)
  y = tf.reduce_max(X, axis=-1)
  
  initial_learning_rate = 0.003
  act, units, depth = 'relu', 16, 3
  
  inp = tf.keras.layers.Input(dim)
  x = inp
  
  for _ in range(depth):
      x = tf.keras.layers.Dense(units, activation=act)(x)
      x = tf.keras.layers.BatchNormalization()(x)
  
  x = tf.keras.layers.Dense(int((dim * units)**0.5), activation="elu")(x)
  x = tf.keras.layers.BatchNormalization()(x)
  
  x = tf.keras.layers.Dense(dim, activation="softmax")(x)
  
  # Skip this step to make categorical predictions
  x = tf.reduce_sum(inp * x, axis=-1, keepdims=True)
  
  model = tf.keras.models.Model(inp, x)
  model.summary()
  
  is_categorical = model.output.shape[1] == dim
  
  model.compile(
      optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
      loss=tf.keras.losses.CategoricalCrossentropy() if is_categorical else
           tf.keras.losses.MeanAbsolutePercentageError(),
      metrics=[tf.keras.metrics.CategoricalAccuracy()] if is_categorical else []
  )
  
  stopping_callback = tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True)
  reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor="loss", factor=0.1**0.5, patience=5,
                                                            verbose=1, min_lr=1e-4)
  
  target = tf.cast(X == y[:,None], tf.float32) if is_categorical else y
  
  h = model.fit(X, target, verbose=1,
                batch_size=1024, epochs=10000, validation_split=validation_split,
                callbacks=[stopping_callback, reduce_lr_callback])
  
  n_val = int(n * validation_split)
  print([model.evaluate(X[:n_val],  target[:n_val]),
         model.evaluate(X[-n_val:], target[-n_val:])])
  
  ```
  
  An alternative formulation outputs just the softmax activation, this obtains an accuracy of 98 - 99%. Granted, it isn't 100% accurate.
  
  Summary of the model with just one output:
  
  ```
  __________________________________________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_273 (InputLayer)          [(None, 5)]          0                                            
  __________________________________________________________________________________________________
  dense_868 (Dense)               (None, 16)           96          input_273[0][0]                  
  __________________________________________________________________________________________________
  batch_normalization_577 (BatchN (None, 16)           64          dense_868[0][0]                  
  __________________________________________________________________________________________________
  dense_869 (Dense)               (None, 16)           272         batch_normalization_577[0][0]    
  __________________________________________________________________________________________________
  batch_normalization_578 (BatchN (None, 16)           64          dense_869[0][0]                  
  __________________________________________________________________________________________________
  dense_870 (Dense)               (None, 16)           272         batch_normalization_578[0][0]    
  __________________________________________________________________________________________________
  batch_normalization_579 (BatchN (None, 16)           64          dense_870[0][0]                  
  __________________________________________________________________________________________________
  dense_871 (Dense)               (None, 8)            136         batch_normalization_579[0][0]    
  __________________________________________________________________________________________________
  batch_normalization_580 (BatchN (None, 8)            32          dense_871[0][0]                  
  __________________________________________________________________________________________________
  dense_872 (Dense)               (None, 5)            45          batch_normalization_580[0][0]    
  __________________________________________________________________________________________________
  tf_op_layer_mul_251 (TensorFlow [(None, 5)]          0           input_273[0][0]                  
                                                                   dense_872[0][0]                  
  __________________________________________________________________________________________________
  tf_op_layer_Sum_329 (TensorFlow [(None, 1)]          0           tf_op_layer_mul_251[0][0]        
  ==================================================================================================
  Total params: 1,045
  Trainable params: 933
  Non-trainable params: 112
  __________________________________________________________________________________________________
  
  ```
  
  I don't know what kind your original problem is, but maybe you could do some feature-engineering by supplying the min/max/median directly to the network. Then the network wouldn't need to learn to approximate those functions.
  
  > **Comments:**
  > **username_0**: Thank you! The secret here seems to be this layer: x = tf.reduce_sum(inp * x, axis=-1, keepdims=True) Can you give any intuition or guidance as to why you decided to do it this way? The elu activation layer with a very specific number of units is also interesting, can you shed some light on that? And the idea of putting in the min/max/median, I don't think I've seen that anywhere before?  
  > **username_0**: I also noticed this doesn't seem to extrapolate well to other functions - e.g. if we use reduce_min() instead of reduce_max(), it doesn't work at all  
  > **username_0**: For reference, one of the most simple functions I actually care about approximating is something like this:   https://gist.github.com/JustASquid/376ecc8f6469fb5a4262941c293352ab  
  > **username_3**: The `tf.reduce_sum(inp * x, axis=-1, keepdims=True)` came from a hunch that maybe the `softmax` would learn to pick the correct input, and push the weight of the others to zero. It is surprising that this didn't work when the aggregation is `min`! However the categorical prediction works on both cases, but clearly you'd like to solve this as a regression problem instead. I'll post an update if something comes to my mind.  
  > **username_3**: The number of dimensions in `elu` is the geometric mean of its input and output size, I have a habbit of doing that. And I have found that `elu` works better in regression tasks than `relu`, actually it can approximate `relu` (or a plain linear function) arbitrary well when combined with batch normalization and subsequent layers.  
  > **username_3**: Actually in addition to just min/max/median, if you input is large (like tens or hundreds of samples) then you should try augmenting the input with 0th, 10th, 20th, ...50th, ... 90th and 100th percentiles of each row. It is nice that these statistics are invariant to the order of input samples, and hopefully the network will learn to use them to predict the correct output.  
  > **username_0**: it's actually potentially feasible for my application to do it with a categorical selection. However, your solution with an accuracy of 98-99% is also not acceptable, I need an accuracy of 99.99%+ before I could consider this to be feasible.  
  > **username_3**: I think the classification task is unfeasible to solve accurately and reliably with a neural network, since the correct output is very sensitive to the inputs. Like having inputs `0, 0, 0.99, 1.01, 0` has a different correct answer than `0, 0, 1.01, 0.99, 0` (this can be generalized to any epsilon).  
  > **username_0**: That's a good perspective. I would have to imagine it's possible to design some architecture/layer that deals with these cases better? I wonder if there's anything in the literature about this?  
  > **username_3**: I think it is very difficult, wihtout having explicit min/max/percentile inputs to the network. And even then, a regression network might learn that the correct output is `max(input) + 1e-3 * x[0] - 1e-5 * x[1]` etc. which is always incorrect by some amount.  
  > **username_0**: maybe something closer to a decision tree would work better? If I can find something differentiable then it could still work for my application...  
  
  ---
  
  ### Answer by username_1 on 2023-01-09. Score: 0
  
  Assuming the $N$ numbers are different, a function that combines sums, products and the hard step function for solving the maximum may look like this.
  
  $$ max(x\_1,x\_2,...,x\_N) = \sum\_{i=1}^N x\_i \prod\_{j=1,j\neq i}^N sign(x\_i-x\_j) $$
  
  $sign(z)$ is $1$ for $z>0$ and $0$ for $z<0$.
  
  > **Comments:**
  > **username_0**: The example is just to demonstrate regressing some unknown function of N inputs with 1 output. You can imagine replacing max() with something else (e.g. sum() ) and the problem is still valid.  
  
  ---
  
  