Restrict output range of a neuron based on output of other neurons I have a neural network with three output neurons $X_1$, $X_2$ and $X_3$ with output range in [-1, 1].

I have many training data split in 80:20 ratio between training & testing sets. While looking at the output of the neural network, I noticed a big thing that the neural network haven't learned.
There is a restriction in the output as $X_1 < X_2 < X_3$.
I found that only with training, I can't achieve this restriction. Is there a recommended way theoretically or using Keras library to tell the neural network about this restriction.

You can enforce the constraint with an additional transformation. Suppose we have $x_i = \sigma(AH + b)$ where $\sigma$ is the sigmoid activation and $A, b$ are the weights and biases.
We can introduce a fixed matrix
$W = \begin{bmatrix}1 & 0 & 0 \\ 1 & 1 & 0 \\ 1 & 1 & 1\end{bmatrix}$, which is not updated during back-propagation (that is, it is non-trainable). We know that $Wx$ will satisfy the constraint because $x_i$ are all positive, so we have $0 < x_1 < x_1 + x_2 < x_1 + x_2 + x_3 < 3$.
If you need the result to be in $[-1,1]$, then you can just rescale and shift:
$\frac{2}{3}Wx - 1$. The choice of $\frac{2}{3}$ means that the largest value possible is 2, and the smallest possible value remains 0. Shifting by $-1$ moves the interval from $[0,2]$ to $[-1,1]$.
