Answers.Body: <p>Let's first express a network of arbitrary topology and heterogeneous or homogeneous cell type arrangements as</p>
  
  <p><span class="math-container">$$ N(T, H, s) := \, \big[\, \mathcal{Y} = F(P_s, \, \mathcal{X}) \,\big] \\
     s \in \mathbb{C} \; \land \; s \le S \; \text{,} $$</span></p>
  
  <p>where <span class="math-container">$S$</span> is the number of learning states or rounds, <span class="math-container">$N$</span> is the network of <span class="math-container">$T$</span> topology and <span class="math-container">$H$</span> hyper-parameter structure and values that at stage <span class="math-container">$s$</span> produces a <span class="math-container">$P$</span> parameterized function <span class="math-container">$f$</span> of <span class="math-container">$\mathcal{X}$</span> resulting in <span class="math-container">$\mathcal{Y}$</span>. In supervised learning, the goal is that <span class="math-container">$F(P_s)$</span> approaches a conceptually ideal function <span class="math-container">$F_i$</span> as <span class="math-container">$s \rightarrow S$</span>.</p>
  
  <p>The popular loss aggregation norms are not quite as the question defines them. The below more canonically expresses the level 1 and 2 norms, which systematically aggregate multidimensional disparity between an intermediate result at some stage (epoch and example index) of training and the conceptual ideal toward which the network in training is intended to converge.</p>
  
  <p><span class="math-container">$$ {||F-\mathcal{Y}||}_1 = \sum{|F_i - y_i|} \\
     {||F-\mathcal{Y}||}_2 = \sqrt{\sum{(F_i - y_i)}^2} $$</span></p>
  
  <p>These equations have been mutated by various authors to make various points, but those mutations have obscured the obviousness of their original relationship. The first is where distance can be aggregated through only orthogonal vector displacements. The second is where aggregation uses the minimum Cartesian distance by extending the Pythagorean theorem.</p>
  
  <p>Note that <em>quadratic loss</em> is a term with some ambiguity. These are all broadly describable as quadratic expressions of loss.</p>
  
  <ul>
  <li>Distance as described in the second expression above</li>
  <li>RMS where the sum is divided by the number of dimensions
  <span class="math-container">$\text{count}(i)$</span></li>
  <li>Just the sum of squared difference by itself</li>
  </ul>
  
  <p>Cross entropy is an extension of Claude Shannon's information theory concepts based on the work of Bohr, Boltzmann, Gibbs, Maxwell, von Neumann, Frisch, Fermi, and others who were interested in quanta and the thermodynamic concept of entropy as a universal principle running through mater, energy, and knowledge.</p>
  
  <p><span class="math-container">$$ S = k_B \log{\Omega} \\
     H(X) = - \sum_i p(x_i) \, \log_2{\, p(x_i)} \\
     H(p, \, q) = -\sum_{x \in \mathcal{X}} \, p(x) \, \log_2{\, q(x)} $$</span></p>
  
  <p>In this progression of theory, we begin with a fundamental postulate in quantum physics, where <span class="math-container">$k_B$</span> is Boltzmann's constant and <span class="math-container">$\Omega$</span> are the number of microstates for the quanta. The next relation is Shannon's adaptation for information, where <span class="math-container">$H$</span> is the entropy in bits, thus the <span class="math-container">$\log_2$</span> instead of a natural logarithm. The third relation above expresses cross-entropy in bits for features <span class="math-container">$\mathcal{X}$</span> is based on the Kullback-Leibler divergence.  the p-attenuated sum of bits of q-information in .</p>
  
  <p>Notice that <span class="math-container">$p$</span> and <span class="math-container">$q$</span> are probabilities, not <span class="math-container">$F$</span> or <span class="math-container">$\mathcal{Y}$</span> values, so one cannot substitute labels and outputs of a network into them and retain the meaning of cross entropy. Therefore level 1 and 2 norms are closely related, but cross-entropy is not a norm; it is the dispersion of one thing Cartesian distance aggregation scheme like them. Cross-entropy is remotely related but is statistically more sophisticated. To produce a cross-entropy loss function of form</p>
  
  <p><span class="math-container">$$ {||F-\mathcal{Y}||}_H = \mathcal{P}(F, y) \; \text{,} $$</span></p>
  
  <p>one must derive the probabilistic function <span class="math-container">$\mathcal{P}$</span> that represents the cross entropy for two distributions in some way that is theoretically sound on the basis of both information theory and convergence resource conservation. It is not clear that the interpretation of cross entropy in the context of gradient descent and back propagation has caught up with the concepts of entropy in quantum theory. That's an area needing further research and deeper theoretical consideration.</p>
  
  <p>In the question, the cross-entropy expression is not properly characterized, most evident in the fact that the expression is independent of the labels <span class="math-container">$\mathcal{Y}$</span>, which would be fine if for unsupervised learning except that no other basis for evaluation is represented in the expression. For the term cross-entropy to be valid, the basis for evaluation must include two distributions, a target one and one that represents the current state of learning.</p>
  
  <p>The derivatives of the three norms (assuming the cross entropy is properly characterized) can be studied for the case of <span class="math-container">$\ell$</span> ReLU layers by generalizing the chain rule (from differential calculus) as applied to ReLU and the loss function developed by applying each of the three norms to aggregate measures of disparity from optimal.</p>
  
  <p>Regarding the inference in sub-question (1) nothing of particular value can be assumed about the Jacobians of level 2 norms from level 1 norms, both with respect to parameters <span class="math-container">$P$</span> or vice versa, except the retention of sign. This is because we cannot determine much about the correlation between the output channels of the network.</p>
  
  <p>There is no doubt, regarding sub-question (2), that some constraint, set of constraints, stochastic distribution applied to initialization, hyper-parameter settings, or data set features, labels, or number of examples have implications for the reliability and accuracy of convergence. The PAC (probably approximately correct) learning framework is one system of theory that approaches this question with mathematical rigor. One of its practical uses, among others, is to derive inequalities that predict feasibility in some cases and produce more lucid approaches to learning system projects.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-16T15:49:58.077Z
Answers.LastActivityDate: 2019-01-17T04:00:36.763Z
Answers.LastEditDate: 2019-01-17T04:00:36.763Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I have a Deep Feedforward Neural Network <span class="math-container">$F: W \times \mathbb{R}^d \rightarrow \mathbb{R}^k$</span> (where <span class="math-container">$W$</span> is the space of the weights) with <span class="math-container">$L$</span> hidden layers, <span class="math-container">$m$</span> neurones per layer and ReLu activation. The output layer has a softmax activation function.</p>
  
  <p>I can consider two different loss functions: </p>
  
  <p><span class="math-container">$L_1 = \frac{1}{2} \sum_i || F(W,x_i) - y||^2$</span>  <span class="math-container">$
   \ \ \ $</span> and   <span class="math-container">$\ \ \ L_2  = -\sum_i log(F(w,x_i)_{y_i})$</span></p>
  
  <p>where the first one is the classic quadratic loss and the second one is cross entropy loss. </p>
  
  <p>I'd like to study the norm of the derivative of the loss function and see how the two are related, which means: </p>
  
  <p>1) Let's assume I know that <span class="math-container">$|| \frac{\partial L_2(W, x_i)}{\partial W}|| &gt; r$</span>, where <span class="math-container">$r$</span> is a small constant. What can I assume about <span class="math-container">$|| \frac{\partial L_1(W, x_i)}{\partial W}||$</span> ?</p>
  
  <p>2) Are there any result which tell you that, under some hypothesis (even strict ones) such as a specific random initialisation,  <span class="math-container">$|| \frac{\partial L_1(W, x_i)}{\partial W}||$</span> doesn't go to zero   during training?</p>
  
  <p>Thank you</p>
  
Tags: <neural-networks><feedforward-neural-networks><objective-functions>
Title: Comparing and studying Loss Functions
UserDisplayName: Alfred
ThreadText: # Comparing and studying Loss Functions
  
  (Asked by: username_0 on 2019-01-16)
  
  I have a Deep Feedforward Neural Network $F: W \times \mathbb{R}^d \rightarrow \mathbb{R}^k$ (where $W$ is the space of the weights) with $L$ hidden layers, $m$ neurones per layer and ReLu activation. The output layer has a softmax activation function.
  
  I can consider two different loss functions:
  
  $L\_1 = \frac{1}{2} \sum\_i || F(W,x\_i) - y||^2$ $
  \ \ \ $ and $\ \ \ L\_2 = -\sum\_i log(F(w,x\_i)\_{y\_i})$
  
  where the first one is the classic quadratic loss and the second one is cross entropy loss.
  
  I'd like to study the norm of the derivative of the loss function and see how the two are related, which means:
  
  1) Let's assume I know that $|| \frac{\partial L\_2(W, x\_i)}{\partial W}|| > r$, where $r$ is a small constant. What can I assume about $|| \frac{\partial L\_1(W, x\_i)}{\partial W}||$ ?
  
  2) Are there any result which tell you that, under some hypothesis (even strict ones) such as a specific random initialisation, $|| \frac{\partial L\_1(W, x\_i)}{\partial W}||$ doesn't go to zero during training?
  
  Thank you
  
  **Tags:** neural-networks, feedforward-neural-networks, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-16. Score: -1 | Accepted answer
  
  Let's first express a network of arbitrary topology and heterogeneous or homogeneous cell type arrangements as
  
  $$ N(T, H, s) := \, \big[\, \mathcal{Y} = F(P\_s, \, \mathcal{X}) \,\big] \\
  s \in \mathbb{C} \; \land \; s \le S \; \text{,} $$
  
  where $S$ is the number of learning states or rounds, $N$ is the network of $T$ topology and $H$ hyper-parameter structure and values that at stage $s$ produces a $P$ parameterized function $f$ of $\mathcal{X}$ resulting in $\mathcal{Y}$. In supervised learning, the goal is that $F(P\_s)$ approaches a conceptually ideal function $F\_i$ as $s \rightarrow S$.
  
  The popular loss aggregation norms are not quite as the question defines them. The below more canonically expresses the level 1 and 2 norms, which systematically aggregate multidimensional disparity between an intermediate result at some stage (epoch and example index) of training and the conceptual ideal toward which the network in training is intended to converge.
  
  $$ {||F-\mathcal{Y}||}\_1 = \sum{|F\_i - y\_i|} \\
  {||F-\mathcal{Y}||}\_2 = \sqrt{\sum{(F\_i - y\_i)}^2} $$
  
  These equations have been mutated by various authors to make various points, but those mutations have obscured the obviousness of their original relationship. The first is where distance can be aggregated through only orthogonal vector displacements. The second is where aggregation uses the minimum Cartesian distance by extending the Pythagorean theorem.
  
  Note that *quadratic loss* is a term with some ambiguity. These are all broadly describable as quadratic expressions of loss.
  
  * Distance as described in the second expression above
  * RMS where the sum is divided by the number of dimensions
    $\text{count}(i)$
  * Just the sum of squared difference by itself
  
  Cross entropy is an extension of Claude Shannon's information theory concepts based on the work of Bohr, Boltzmann, Gibbs, Maxwell, von Neumann, Frisch, Fermi, and others who were interested in quanta and the thermodynamic concept of entropy as a universal principle running through mater, energy, and knowledge.
  
  $$ S = k\_B \log{\Omega} \\
  H(X) = - \sum\_i p(x\_i) \, \log\_2{\, p(x\_i)} \\
  H(p, \, q) = -\sum\_{x \in \mathcal{X}} \, p(x) \, \log\_2{\, q(x)} $$
  
  In this progression of theory, we begin with a fundamental postulate in quantum physics, where $k\_B$ is Boltzmann's constant and $\Omega$ are the number of microstates for the quanta. The next relation is Shannon's adaptation for information, where $H$ is the entropy in bits, thus the $\log\_2$ instead of a natural logarithm. The third relation above expresses cross-entropy in bits for features $\mathcal{X}$ is based on the Kullback-Leibler divergence. the p-attenuated sum of bits of q-information in .
  
  Notice that $p$ and $q$ are probabilities, not $F$ or $\mathcal{Y}$ values, so one cannot substitute labels and outputs of a network into them and retain the meaning of cross entropy. Therefore level 1 and 2 norms are closely related, but cross-entropy is not a norm; it is the dispersion of one thing Cartesian distance aggregation scheme like them. Cross-entropy is remotely related but is statistically more sophisticated. To produce a cross-entropy loss function of form
  
  $$ {||F-\mathcal{Y}||}\_H = \mathcal{P}(F, y) \; \text{,} $$
  
  one must derive the probabilistic function $\mathcal{P}$ that represents the cross entropy for two distributions in some way that is theoretically sound on the basis of both information theory and convergence resource conservation. It is not clear that the interpretation of cross entropy in the context of gradient descent and back propagation has caught up with the concepts of entropy in quantum theory. That's an area needing further research and deeper theoretical consideration.
  
  In the question, the cross-entropy expression is not properly characterized, most evident in the fact that the expression is independent of the labels $\mathcal{Y}$, which would be fine if for unsupervised learning except that no other basis for evaluation is represented in the expression. For the term cross-entropy to be valid, the basis for evaluation must include two distributions, a target one and one that represents the current state of learning.
  
  The derivatives of the three norms (assuming the cross entropy is properly characterized) can be studied for the case of $\ell$ ReLU layers by generalizing the chain rule (from differential calculus) as applied to ReLU and the loss function developed by applying each of the three norms to aggregate measures of disparity from optimal.
  
  Regarding the inference in sub-question (1) nothing of particular value can be assumed about the Jacobians of level 2 norms from level 1 norms, both with respect to parameters $P$ or vice versa, except the retention of sign. This is because we cannot determine much about the correlation between the output channels of the network.
  
  There is no doubt, regarding sub-question (2), that some constraint, set of constraints, stochastic distribution applied to initialization, hyper-parameter settings, or data set features, labels, or number of examples have implications for the reliability and accuracy of convergence. The PAC (probably approximately correct) learning framework is one system of theory that approaches this question with mathematical rigor. One of its practical uses, among others, is to derive inequalities that predict feasibility in some cases and produce more lucid approaches to learning system projects.
  
  
  ---
  
  