<p>An interesting problem. This network has only 933 trainable parameters, and obtains <code>MeanAbsolutePercentageError</code> of 0.01 - 0.04. It is based on a softmax activation, to choose which item from the input to choose.</p>
<pre><code>n, dim, validation_split = 100000, 5, 0.1
X = tf.random.uniform(shape=(n, dim), minval=0.0, maxval=1.0)
y = tf.reduce_max(X, axis=-1)

initial_learning_rate = 0.003
act, units, depth = 'relu', 16, 3

inp = tf.keras.layers.Input(dim)
x = inp

for _ in range(depth):
    x = tf.keras.layers.Dense(units, activation=act)(x)
    x = tf.keras.layers.BatchNormalization()(x)

x = tf.keras.layers.Dense(int((dim * units)**0.5), activation=&quot;elu&quot;)(x)
x = tf.keras.layers.BatchNormalization()(x)

x = tf.keras.layers.Dense(dim, activation=&quot;softmax&quot;)(x)

# Skip this step to make categorical predictions
x = tf.reduce_sum(inp * x, axis=-1, keepdims=True)

model = tf.keras.models.Model(inp, x)
model.summary()

is_categorical = model.output.shape[1] == dim

model.compile(
    optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
    loss=tf.keras.losses.CategoricalCrossentropy() if is_categorical else
         tf.keras.losses.MeanAbsolutePercentageError(),
    metrics=[tf.keras.metrics.CategoricalAccuracy()] if is_categorical else []
)

stopping_callback = tf.keras.callbacks.EarlyStopping(monitor=&quot;val_loss&quot;, patience=10, restore_best_weights=True)
reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor=&quot;loss&quot;, factor=0.1**0.5, patience=5,
                                                          verbose=1, min_lr=1e-4)

target = tf.cast(X == y[:,None], tf.float32) if is_categorical else y

h = model.fit(X, target, verbose=1,
              batch_size=1024, epochs=10000, validation_split=validation_split,
              callbacks=[stopping_callback, reduce_lr_callback])

n_val = int(n * validation_split)
print([model.evaluate(X[:n_val],  target[:n_val]),
       model.evaluate(X[-n_val:], target[-n_val:])])
</code></pre>
<p>An alternative formulation outputs just the softmax activation, this obtains an accuracy of 98 - 99%. Granted, it isn't 100% accurate.</p>
<p>Summary of the model with just one output:</p>
<pre><code>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_273 (InputLayer)          [(None, 5)]          0                                            
__________________________________________________________________________________________________
dense_868 (Dense)               (None, 16)           96          input_273[0][0]                  
__________________________________________________________________________________________________
batch_normalization_577 (BatchN (None, 16)           64          dense_868[0][0]                  
__________________________________________________________________________________________________
dense_869 (Dense)               (None, 16)           272         batch_normalization_577[0][0]    
__________________________________________________________________________________________________
batch_normalization_578 (BatchN (None, 16)           64          dense_869[0][0]                  
__________________________________________________________________________________________________
dense_870 (Dense)               (None, 16)           272         batch_normalization_578[0][0]    
__________________________________________________________________________________________________
batch_normalization_579 (BatchN (None, 16)           64          dense_870[0][0]                  
__________________________________________________________________________________________________
dense_871 (Dense)               (None, 8)            136         batch_normalization_579[0][0]    
__________________________________________________________________________________________________
batch_normalization_580 (BatchN (None, 8)            32          dense_871[0][0]                  
__________________________________________________________________________________________________
dense_872 (Dense)               (None, 5)            45          batch_normalization_580[0][0]    
__________________________________________________________________________________________________
tf_op_layer_mul_251 (TensorFlow [(None, 5)]          0           input_273[0][0]                  
                                                                 dense_872[0][0]                  
__________________________________________________________________________________________________
tf_op_layer_Sum_329 (TensorFlow [(None, 1)]          0           tf_op_layer_mul_251[0][0]        
==================================================================================================
Total params: 1,045
Trainable params: 933
Non-trainable params: 112
__________________________________________________________________________________________________
</code></pre>
<p>I don't know what kind your original problem is, but maybe you could do some feature-engineering by supplying the min/max/median directly to the network. Then the network wouldn't need to learn to approximate those functions.</p>

CC BY-SA 4.0
2023-01-09T22:31:40.713Z
Thank you! The secret here seems to be this layer:
x = tf.reduce_sum(inp * x, axis=-1, keepdims=True)
Can you give any intuition or guidance as to why you decided to do it this way?
The elu activation layer with a very specific number of units is also interesting, can you shed some light on that? And the idea of putting in the min/max/median, I don't think I've seen that anywhere before?
Daniel
CC BY-SA 4.0
2023-01-09T23:38:22.073Z
I also noticed this doesn't seem to extrapolate well to other functions - e.g. if we use reduce_min() instead of reduce_max(), it doesn't work at all
Daniel
CC BY-SA 4.0
2023-01-10T00:31:33.990Z
For reference, one of the most simple functions I actually care about approximating is something like this: 

https://gist.github.com/JustASquid/376ecc8f6469fb5a4262941c293352ab
Daniel
CC BY-SA 4.0
2023-01-10T20:57:32.440Z
The `tf.reduce_sum(inp * x, axis=-1, keepdims=True)` came from a hunch that maybe the `softmax` would learn to pick the correct input, and push the weight of the others to zero. It is surprising that this didn't work when the aggregation is `min`! However the categorical prediction works on both cases, but clearly you'd like to solve this as a regression problem instead. I'll post an update if something comes to my mind.
NikoNyrh
CC BY-SA 4.0
2023-01-10T20:59:20.367Z
The number of dimensions in `elu` is the geometric mean of its input and output size, I have a habbit of doing that. And I have found that `elu` works better in regression tasks than `relu`, actually it can approximate `relu` (or a plain linear function) arbitrary well when combined with batch normalization and subsequent layers.
NikoNyrh
CC BY-SA 4.0
2023-01-10T21:02:50.680Z
Actually in addition to just min/max/median, if you input is large (like tens or hundreds of samples) then you should try augmenting the input with 0th, 10th, 20th, ...50th, ... 90th and 100th percentiles of each row. It is nice that these statistics are invariant to the order of input samples, and hopefully the network will learn to use them to predict the correct output.
NikoNyrh
CC BY-SA 4.0
2023-01-11T01:53:04.680Z
it's actually potentially feasible for my application to do it with a categorical selection. However, your solution with an accuracy of 98-99% is also not acceptable, I need an accuracy of 99.99%+ before I could consider this to be feasible.
Daniel
CC BY-SA 4.0
2023-01-14T21:33:53.260Z
I think the classification task is unfeasible to solve accurately and reliably with a neural network, since the correct output is very sensitive to the inputs. Like having inputs `0, 0, 0.99, 1.01, 0` has a different correct answer than `0, 0, 1.01, 0.99, 0` (this can be generalized to any epsilon).
NikoNyrh
CC BY-SA 4.0
2023-01-14T21:39:43.970Z
That's a good perspective. I would have to imagine it's possible to design some architecture/layer that deals with these cases better? I wonder if there's anything in the literature about this?
Daniel
CC BY-SA 4.0
2023-01-14T21:53:29.750Z
I think it is very difficult, wihtout having explicit min/max/percentile inputs to the network. And even then, a regression network might learn that the correct output is `max(input) + 1e-3 * x[0] - 1e-5 * x[1]` etc. which is always incorrect by some amount.
NikoNyrh
CC BY-SA 4.0
2023-01-14T22:37:50.437Z
maybe something closer to a decision tree would work better? If I can find something differentiable then it could still work for my application...
Daniel
CC BY-SA 4.0
2023-01-09T18:12:15.050Z
2023-01-09T18:12:15.050Z
NikoNyrh
<p>Assuming the <span class="math-container">$N$</span> numbers are different, a function that combines sums, products and the hard step function for solving the maximum may look like this.</p>
<p><span class="math-container">$$ max(x_1,x_2,...,x_N) = \sum_{i=1}^N x_i \prod_{j=1,j\neq i}^N sign(x_i-x_j) $$</span></p>
<p><span class="math-container">$sign(z)$</span> is <span class="math-container">$1$</span> for <span class="math-container">$z&gt;0$</span> and <span class="math-container">$0$</span> for <span class="math-container">$z&lt;0$</span>.</p>

CC BY-SA 4.0
2023-01-09T22:34:20.620Z
The example is just to demonstrate regressing some unknown function of N inputs with 1 output. You can imagine replacing max() with something else (e.g. sum() ) and the problem is still valid.
Daniel
CC BY-SA 4.0
2023-01-09T12:32:06.643Z
2023-01-09T12:32:06.643Z
Jaume Oliver Lafont
<p>Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm facing.</p>
<p>One could attempt to train an MLP to solve this problem using randomly generated input data. The following code is an example in Keras/Tensorflow. It generates 1000000 random examples (A large number, so we can reduce overfitting as a factor). It also applies some simple techniques like LR decay and early stopping to optimize training.</p>
<pre><code>import tensorflow as tf

# Create a dataset where the y value is the largest of 5 random x values.
x = tf.random.uniform(shape=(1000000, 5), minval=0.0, maxval=1.0)
y = tf.reduce_max(x, axis=-1)

dataset = tf.data.Dataset.from_tensor_slices(
    (x, y)
).shuffle(1000000).batch(1024).prefetch(1)

# See the post for discussion of hyperparameter optimization
units = 400
depth = 5
initial_learning_rate = 0.001

model = tf.keras.models.Sequential()

for i in range(depth):
    model.add(tf.keras.layers.Dense(units, activation=&quot;relu&quot;))
model.add(tf.keras.layers.Dense(1))

model.compile(
    optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
    loss=tf.keras.losses.MeanAbsolutePercentageError(),
)

stopping_callback = tf.keras.callbacks.EarlyStopping(monitor=&quot;loss&quot;, patience=5)

reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor=&quot;loss&quot;, factor=0.75, patience=2)

model.fit(dataset, epochs=1000)
</code></pre>
<p>The metric we care about is the mean absolute percentage error. When we run the code, the model trains successfully. But very importantly, the loss does not ever go below ~0.2 with these hyperparameters.</p>
<p>0.2% error sounds pretty good, but it's actually unacceptable for my use case. I need much more precision, as in practice the model is used in a situation where any errors are magnified.</p>
<p>At this point, you may want to suggest hyperparameter tuning, but I assure you I have done an extensive amount of hyperparameter tuning on this and similar problems.</p>
<p>The model is clearly underfitting, and it is possible to decrease the error by increasing the number of parameters in the network. Empirically, I've noticed that increasing the number of parameters (and number of examples) by an order of magnitude can reduce the error by a similar order of magnitude.</p>
<p>However, intuitively this just feels wrong. It is not practical to have a network with millions, or even tens of millions of parameters, just to regress such a simple function. It feels like there should be an architecture that can learn a simple function much more efficiently, but I have not been able to find any reference to such an alternative architecture.</p>
<p>Again, of course this particular example is trivial, but hopefully you can imagine a real-world analogue where an unknown (more complex, but similar) function is expressed by a similar dataset. And in such a real-world example, the unacceptable error still persists (and is much more pronounced, for reasons that I don't fully understand).</p>
<p>Does anyone have any idea what's going on here? Am I missing something? How can I improve the performance to a place that would be acceptable?</p>

CC BY-SA 4.0
2023-01-09T06:26:54.657Z
Would softmax (https://en.wikipedia.org/wiki/Softmax_function) help?
Jaume Oliver Lafont
CC BY-SA 4.0
2023-01-09T06:28:22.537Z
@JaumeOliverLafont Not in practice, as far as I can recall. Is there theory behind why softmax would be better than relu here?
Daniel
CC BY-SA 4.0
2023-01-09T10:10:57.690Z
Any reason why you would not model 5 output nodes and have them each score a percentage of how likely it is that that one is the largest?
Robin van Hoorn
<tensorflow><keras><regression><multilayer-perceptrons>
Why is a simple regression problem so hard for an MLP to learn?
Daniel
# Why is a simple regression problem so hard for an MLP to learn?

(Asked by: username_0 on 2023-01-09)

Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm facing.
One could attempt to train an MLP to solve this problem using randomly generated input data. The following code is an example in Keras/Tensorflow. It generates 1000000 random examples (A large number, so we can reduce overfitting as a factor). It also applies some simple techniques like LR decay and early stopping to optimize training.
import tensorflow as tf

# Create a dataset where the y value is the largest of 5 random x values.
x = tf.random.uniform(shape=(1000000, 5), minval=0.0, maxval=1.0)
y = tf.reduce_max(x, axis=-1)

dataset = tf.data.Dataset.from_tensor_slices(
    (x, y)
).shuffle(1000000).batch(1024).prefetch(1)

# See the post for discussion of hyperparameter optimization
units = 400
depth = 5
initial_learning_rate = 0.001

model = tf.keras.models.Sequential()

for i in range(depth):
    model.add(tf.keras.layers.Dense(units, activation="relu"))
model.add(tf.keras.layers.Dense(1))

model.compile(
    optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
    loss=tf.keras.losses.MeanAbsolutePercentageError(),
)

stopping_callback = tf.keras.callbacks.EarlyStopping(monitor="loss", patience=5)

reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor="loss", factor=0.75, patience=2)

model.fit(dataset, epochs=1000)

The metric we care about is the mean absolute percentage error. When we run the code, the model trains successfully. But very importantly, the loss does not ever go below ~0.2 with these hyperparameters.
0.2% error sounds pretty good, but it's actually unacceptable for my use case. I need much more precision, as in practice the model is used in a situation where any errors are magnified.
At this point, you may want to suggest hyperparameter tuning, but I assure you I have done an extensive amount of hyperparameter tuning on this and similar problems.
The model is clearly underfitting, and it is possible to decrease the error by increasing the number of parameters in the network. Empirically, I've noticed that increasing the number of parameters (and number of examples) by an order of magnitude can reduce the error by a similar order of magnitude.
However, intuitively this just feels wrong. It is not practical to have a network with millions, or even tens of millions of parameters, just to regress such a simple function. It feels like there should be an architecture that can learn a simple function much more efficiently, but I have not been able to find any reference to such an alternative architecture.
Again, of course this particular example is trivial, but hopefully you can imagine a real-world analogue where an unknown (more complex, but similar) function is expressed by a similar dataset. And in such a real-world example, the unacceptable error still persists (and is much more pronounced, for reasons that I don't fully understand).
Does anyone have any idea what's going on here? Am I missing something? How can I improve the performance to a place that would be acceptable?

**Tags:** tensorflow, keras, regression, multilayer-perceptrons

> **Comments:**
> **username_1**: Would softmax (https://en.wikipedia.org/wiki/Softmax_function) help?  
> **username_0**: @JaumeOliverLafont Not in practice, as far as I can recall. Is there theory behind why softmax would be better than relu here?  
> **username_2**: Any reason why you would not model 5 output nodes and have them each score a percentage of how likely it is that that one is the largest?  

---

## 2 answers

### Answer by username_3 on 2023-01-09. Score: 3

An interesting problem. This network has only 933 trainable parameters, and obtains MeanAbsolutePercentageError of 0.01 - 0.04. It is based on a softmax activation, to choose which item from the input to choose.
n, dim, validation_split = 100000, 5, 0.1
X = tf.random.uniform(shape=(n, dim), minval=0.0, maxval=1.0)
y = tf.reduce_max(X, axis=-1)

initial_learning_rate = 0.003
act, units, depth = 'relu', 16, 3

inp = tf.keras.layers.Input(dim)
x = inp

for _ in range(depth):
    x = tf.keras.layers.Dense(units, activation=act)(x)
    x = tf.keras.layers.BatchNormalization()(x)

x = tf.keras.layers.Dense(int((dim * units)**0.5), activation="elu")(x)
x = tf.keras.layers.BatchNormalization()(x)

x = tf.keras.layers.Dense(dim, activation="softmax")(x)

# Skip this step to make categorical predictions
x = tf.reduce_sum(inp * x, axis=-1, keepdims=True)

model = tf.keras.models.Model(inp, x)
model.summary()

is_categorical = model.output.shape[1] == dim

model.compile(
    optimizer=tf.keras.optimizers.Adam(initial_learning_rate),
    loss=tf.keras.losses.CategoricalCrossentropy() if is_categorical else
         tf.keras.losses.MeanAbsolutePercentageError(),
    metrics=[tf.keras.metrics.CategoricalAccuracy()] if is_categorical else []
)

stopping_callback = tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True)
reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor="loss", factor=0.1**0.5, patience=5,
                                                          verbose=1, min_lr=1e-4)

target = tf.cast(X == y[:,None], tf.float32) if is_categorical else y

h = model.fit(X, target, verbose=1,
              batch_size=1024, epochs=10000, validation_split=validation_split,
              callbacks=[stopping_callback, reduce_lr_callback])

n_val = int(n * validation_split)
print([model.evaluate(X[:n_val],  target[:n_val]),
       model.evaluate(X[-n_val:], target[-n_val:])])

An alternative formulation outputs just the softmax activation, this obtains an accuracy of 98 - 99%. Granted, it isn't 100% accurate.
Summary of the model with just one output:
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_273 (InputLayer)          [(None, 5)]          0                                            
__________________________________________________________________________________________________
dense_868 (Dense)               (None, 16)           96          input_273[0][0]                  
__________________________________________________________________________________________________
batch_normalization_577 (BatchN (None, 16)           64          dense_868[0][0]                  
__________________________________________________________________________________________________
dense_869 (Dense)               (None, 16)           272         batch_normalization_577[0][0]    
__________________________________________________________________________________________________
batch_normalization_578 (BatchN (None, 16)           64          dense_869[0][0]                  
__________________________________________________________________________________________________
dense_870 (Dense)               (None, 16)           272         batch_normalization_578[0][0]    
__________________________________________________________________________________________________
batch_normalization_579 (BatchN (None, 16)           64          dense_870[0][0]                  
__________________________________________________________________________________________________
dense_871 (Dense)               (None, 8)            136         batch_normalization_579[0][0]    
__________________________________________________________________________________________________
batch_normalization_580 (BatchN (None, 8)            32          dense_871[0][0]                  
__________________________________________________________________________________________________
dense_872 (Dense)               (None, 5)            45          batch_normalization_580[0][0]    
__________________________________________________________________________________________________
tf_op_layer_mul_251 (TensorFlow [(None, 5)]          0           input_273[0][0]                  
                                                                 dense_872[0][0]                  
__________________________________________________________________________________________________
tf_op_layer_Sum_329 (TensorFlow [(None, 1)]          0           tf_op_layer_mul_251[0][0]        
==================================================================================================
Total params: 1,045
Trainable params: 933
Non-trainable params: 112
__________________________________________________________________________________________________

I don't know what kind your original problem is, but maybe you could do some feature-engineering by supplying the min/max/median directly to the network. Then the network wouldn't need to learn to approximate those functions.

> **Comments:**
> **username_0**: Thank you! The secret here seems to be this layer: x = tf.reduce_sum(inp * x, axis=-1, keepdims=True) Can you give any intuition or guidance as to why you decided to do it this way? The elu activation layer with a very specific number of units is also interesting, can you shed some light on that? And the idea of putting in the min/max/median, I don't think I've seen that anywhere before?  
> **username_0**: I also noticed this doesn't seem to extrapolate well to other functions - e.g. if we use reduce_min() instead of reduce_max(), it doesn't work at all  
> **username_0**: For reference, one of the most simple functions I actually care about approximating is something like this:   https://gist.github.com/JustASquid/376ecc8f6469fb5a4262941c293352ab  
> **username_3**: The `tf.reduce_sum(inp * x, axis=-1, keepdims=True)` came from a hunch that maybe the `softmax` would learn to pick the correct input, and push the weight of the others to zero. It is surprising that this didn't work when the aggregation is `min`! However the categorical prediction works on both cases, but clearly you'd like to solve this as a regression problem instead. I'll post an update if something comes to my mind.  
> **username_3**: The number of dimensions in `elu` is the geometric mean of its input and output size, I have a habbit of doing that. And I have found that `elu` works better in regression tasks than `relu`, actually it can approximate `relu` (or a plain linear function) arbitrary well when combined with batch normalization and subsequent layers.  
> **username_3**: Actually in addition to just min/max/median, if you input is large (like tens or hundreds of samples) then you should try augmenting the input with 0th, 10th, 20th, ...50th, ... 90th and 100th percentiles of each row. It is nice that these statistics are invariant to the order of input samples, and hopefully the network will learn to use them to predict the correct output.  
> **username_0**: it's actually potentially feasible for my application to do it with a categorical selection. However, your solution with an accuracy of 98-99% is also not acceptable, I need an accuracy of 99.99%+ before I could consider this to be feasible.  
> **username_3**: I think the classification task is unfeasible to solve accurately and reliably with a neural network, since the correct output is very sensitive to the inputs. Like having inputs `0, 0, 0.99, 1.01, 0` has a different correct answer than `0, 0, 1.01, 0.99, 0` (this can be generalized to any epsilon).  
> **username_0**: That's a good perspective. I would have to imagine it's possible to design some architecture/layer that deals with these cases better? I wonder if there's anything in the literature about this?  
> **username_3**: I think it is very difficult, wihtout having explicit min/max/percentile inputs to the network. And even then, a regression network might learn that the correct output is `max(input) + 1e-3 * x[0] - 1e-5 * x[1]` etc. which is always incorrect by some amount.  
> **username_0**: maybe something closer to a decision tree would work better? If I can find something differentiable then it could still work for my application...  

---

### Answer by username_1 on 2023-01-09. Score: 0

Assuming the $N$ numbers are different, a function that combines sums, products and the hard step function for solving the maximum may look like this.

$$ max(x\_1,x\_2,...,x\_N) = \sum\_{i=1}^N x\_i \prod\_{j=1,j\neq i}^N sign(x\_i-x\_j) $$

$sign(z)$ is $1$ for $z>0$ and $0$ for $z<0$.

> **Comments:**
> **username_0**: The example is just to demonstrate regressing some unknown function of N inputs with 1 output. You can imagine replacing max() with something else (e.g. sum() ) and the problem is still valid.  

---

