Answers.Body: <p>Penalty (<a href="https://en.wikipedia.org/wiki/Barrier_function" rel="nofollow noreferrer">barrier function</a>) is perfectly valid and simplest method for simplex type constraint (L1 norm is simplex constraint on absolute values). Any type of barrier function may work, logarithmic, reciprocal or quadratic. All of them supported by any major framework(pytorch, tensorflow), just add them to loss function. You would need some hyperparameter tuning for the scale factor of penalty.</p>
  
  <p>There is more efficient, though more complex way to do it. Instead of putting constraint you can automatically output value wich satisfy simplex constraint:</p>
  
  <p>Assume that L1 norm constraint is <span class="math-container">$\left \|v\right \|_1 \leq 1$</span>, <span class="math-container">$v \in \mathbb{R}^n$</span></p>
  
  <ol>
  <li>put <span class="math-container">$sigmoid(v_i)$</span> activation on output to norm elements to [-1, 1]</li>
  <li>add slack (fake) variable element <span class="math-container">$v_{n+1} = 1 - \sum_{1}^{n}  v_i $</span></li>
  <li>project new <span class="math-container">$v{}'\in \mathbb{R}^{n+1}$</span>, <span class="math-container">$v{}'_i = |v_i|,1\leq  i  \leq n+1$</span> onto unit simplex <a href="https://en.wikipedia.org/wiki/Simplex#Projection_onto_the_standard_simplex" rel="nofollow noreferrer">with standard algorithm</a> (also <a href="https://arxiv.org/abs/1101.6081" rel="nofollow noreferrer">here</a>)</li>
  </ol>
  
  <p>Backpropagation of last step may require <em>differentiable sorting</em>, which is missing in most of frameworks, you may have to look for open sourced implementation, for example extract it from <a href="https://github.com/MaestroGraph/sparse-hyper" rel="nofollow noreferrer">here</a> or use some automatic differentiation package. Both require some substantial code reading/debugging. However in my experience assuming constant <span class="math-container">$\Delta$</span> also works in many cases, in that case  differentiable sorting is not needed. Intuition behind constant <span class="math-container">$\Delta$</span>  is that  <span class="math-container">$\Delta$</span>  could be chosen such way that there is some interval on which it's value doesn't affect sorting order.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-18T08:53:12.107Z
Answers.LastActivityDate: 2019-08-18T09:13:58.340Z
Answers.LastEditDate: 2019-08-18T09:13:58.340Z
Answers.UserDisplayName: mirror2image
Answers.Body: <p>The term 'size' isn't applicable to the tensor output of a network or network.  These are the qualities.</p>
  
  <ul>
  <li>Rank <span class="math-container">$N$</span> that defines the rank of each tensor instance in <span class="math-container">$\mathbb{R}^N$</span></li>
  <li>Ranges of the indices to the dimensions from <span class="math-container">$1$</span> to <span class="math-container">$N$</span></li>
  <li>Ranges of the scalar values that comprise the tensor instance &mdash; If they are discrete rather than real (approximated by floating point or fixed point numbers), then the range is the description of the permissible ordinal values.</li>
  </ul>
  
  <p>The question may be referring to this last quality.</p>
  
  <p>The imposition of a penalty for values that are in the range of the numeric type used as the output of the last activation function but not in the allowable range of output for the desired trained function works in a limited way.  It skews the output distribution with respect to the natural distribution of possible learning states and therefore can easily interfere with convergence quality or speed or both.</p>
  
  <p>There are a number of techniques that map natural output distribution with constrained ranges, but it must be done without skewing the distribution upstream from the technique used, to avoid negatively impacting favorable convergence properties of the artificial network.</p>
  
  <p>One simple case that can be described here is when the number of possible output states is in the set of <span class="math-container">$2^i$</span> where <span class="math-container">$i \in$</span> { positive integers }.  In such a case, the final layer of the network can be <span class="math-container">$i$</span> threshold activation functions with 1 or -1 as possible output values.</p>
  
  <p>In that case, the ordinal then becomes</p>
  
  <p><span class="math-container">$$o = \sum_{n=0}^i \frac {y_n + 1} {2},$$</span></p>
  
  <p>where <span class="math-container">$y_n$</span> is the output from activation function index <span class="math-container">$n$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-26T18:17:14.350Z
Answers.Text: You say "output states is in the set of $2^i$" but the value you describe is $o \in [0,i] $, $o \in \mathbb{Z}$
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-21T19:37:14.457Z
Answers.LastActivityDate: 2018-10-21T19:37:14.457Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>What are some ways to design a neural network with the restriction that the <span class="math-container">$L_1$</span> norm of the output values must be less than or equal to 1? In particular, how would I go about performing back-propagation for this net?</p>
  <p>I was thinking there must be some &quot;penalty&quot; method just like how in the mathematical optimization problem, you can introduce a log barrier function as the &quot;penalty function&quot;</p>
  
Tags: <neural-networks><machine-learning><backpropagation><objective-functions><constrained-optimization>
Title: How do we design a neural network such that the $L_1$ norm of the outputs is less than or equal to 1?
UserDisplayName: Kevvy Kim
ThreadText: # How do we design a neural network such that the $L_1$ norm of the outputs is less than or equal to 1?
  
  (Asked by: username_0 on 2018-10-19)
  
  What are some ways to design a neural network with the restriction that the $L\_1$ norm of the output values must be less than or equal to 1? In particular, how would I go about performing back-propagation for this net?
  
  I was thinking there must be some "penalty" method just like how in the mathematical optimization problem, you can introduce a log barrier function as the "penalty function"
  
  **Tags:** neural-networks, machine-learning, backpropagation, objective-functions, constrained-optimization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-08-18. Score: 2
  
  Penalty ([barrier function](https://en.wikipedia.org/wiki/Barrier_function)) is perfectly valid and simplest method for simplex type constraint (L1 norm is simplex constraint on absolute values). Any type of barrier function may work, logarithmic, reciprocal or quadratic. All of them supported by any major framework(pytorch, tensorflow), just add them to loss function. You would need some hyperparameter tuning for the scale factor of penalty.
  
  There is more efficient, though more complex way to do it. Instead of putting constraint you can automatically output value wich satisfy simplex constraint:
  
  Assume that L1 norm constraint is $\left \|v\right \|\_1 \leq 1$, $v \in \mathbb{R}^n$
  
  1. put $sigmoid(v\_i)$ activation on output to norm elements to [-1, 1]
  2. add slack (fake) variable element $v\_{n+1} = 1 - \sum\_{1}^{n} v\_i $
  3. project new $v{}'\in \mathbb{R}^{n+1}$, $v{}'\_i = |v\_i|,1\leq i \leq n+1$ onto unit simplex [with standard algorithm](https://en.wikipedia.org/wiki/Simplex#Projection_onto_the_standard_simplex) (also [here](https://arxiv.org/abs/1101.6081))
  
  Backpropagation of last step may require *differentiable sorting*, which is missing in most of frameworks, you may have to look for open sourced implementation, for example extract it from [here](https://github.com/MaestroGraph/sparse-hyper) or use some automatic differentiation package. Both require some substantial code reading/debugging. However in my experience assuming constant $\Delta$ also works in many cases, in that case differentiable sorting is not needed. Intuition behind constant $\Delta$ is that $\Delta$ could be chosen such way that there is some interval on which it's value doesn't affect sorting order.
  
  
  ---
  
  ### Answer by username_2 on 2018-10-21. Score: -3
  
  The term 'size' isn't applicable to the tensor output of a network or network. These are the qualities.
  
  * Rank $N$ that defines the rank of each tensor instance in $\mathbb{R}^N$
  * Ranges of the indices to the dimensions from $1$ to $N$
  * Ranges of the scalar values that comprise the tensor instance — If they are discrete rather than real (approximated by floating point or fixed point numbers), then the range is the description of the permissible ordinal values.
  
  The question may be referring to this last quality.
  
  The imposition of a penalty for values that are in the range of the numeric type used as the output of the last activation function but not in the allowable range of output for the desired trained function works in a limited way. It skews the output distribution with respect to the natural distribution of possible learning states and therefore can easily interfere with convergence quality or speed or both.
  
  There are a number of techniques that map natural output distribution with constrained ranges, but it must be done without skewing the distribution upstream from the technique used, to avoid negatively impacting favorable convergence properties of the artificial network.
  
  One simple case that can be described here is when the number of possible output states is in the set of $2^i$ where $i \in$ { positive integers }. In such a case, the final layer of the network can be $i$ threshold activation functions with 1 or -1 as possible output values.
  
  In that case, the ordinal then becomes
  
  $$o = \sum\_{n=0}^i \frac {y\_n + 1} {2},$$
  
  where $y\_n$ is the output from activation function index $n$.
  
  > **Comments:**
  > **username_3**: You say "output states is in the set of $2^i$" but the value you describe is $o \in [0,i] $, $o \in \mathbb{Z}$  
  
  ---
  
  