Answers.Body: <p>Perhaps to qualify as an appropriate generalization of the popular and powerful MLPs, such new types of neural networks need to be able to do stable first order gradient descent optimization which is scalable with big training dataset.</p>
  <p>However, since your proposed operator selector <span class="math-container">$k$</span> is a discrete parameter, optimizing over <span class="math-container">$k$</span> introduces challenges in training, for example, gradient-based methods struggle with discrete optimization. In addition, the interpretability of such networks becomes harder as <span class="math-container">$k$</span> grows, and large <span class="math-container">$k$</span> will obviously introduce numerical instability which impacts convergence of your proposed neural network's training.</p>
  <p>Finally hyperoperators sequence allowed Goodstein to rigorously express and study numerical growth beyond exponentiation (similar to the Ackermann function), and <a href="https://en.wikipedia.org/wiki/Goodstein%27s_theorem" rel="nofollow noreferrer">Goodstein's theorem</a> ultimately highlighted the incompleteness limitation of Peano Arithmetic (PA) which cannot prove that every <a href="https://en.wikipedia.org/wiki/Goodstein%27s_theorem#Goodstein_sequences" rel="nofollow noreferrer">Goodstein sequence</a> eventually terminates at 0, but the same can be proved in the stronger second order arithmetic or ZFC. Therefore the hyperoperators dealing with large number growth are better suited for theoretical edge case studies, but not for the everyday applications target by common ML models. Comparatively speaking, your referenced KANs may have better application since KANs have much less neurons to achieve similar performance of MLPs due to the famous <a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_representation_theorem" rel="nofollow noreferrer">Kolmogorov-Arnold representation theorem</a> linking multivariate/univariate real-valued continuous functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-29T07:48:52.790Z
Answers.Text: The optimizer is not only about gradient descent. It can also be possible to solve it by quantum computing for the optimization problem. Put $k$ and $w$ in superposition so that it explores possible values, and let the system evolve, finding the optimized parameter based on input-output pairs.
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-29T07:53:00.367Z
Answers.Text: I don't see quantum computing is in OP's scope at all, which is yet to become applicable or relevant for everyday ML apps in the near future. Also bear in mind quantum computing as a new and potentially extremely efficient computational technique in some cases don't change computability or incompleteness edge case.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-29T07:56:42.923Z
Answers.Text: Same for the gradient descent, which implies the question has an agnostic optimizer method.
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-29T07:45:58.730Z
Answers.LastActivityDate: 2024-12-29T07:50:07.713Z
Answers.LastEditDate: 2024-12-29T07:50:07.713Z
Answers.UserDisplayName: cinch
Body: <p>We know that artificial neural network (ANN) components are:</p>
  <ul>
  <li><span class="math-container">$x$</span>: input</li>
  <li><span class="math-container">$y$</span>: output</li>
  <li><span class="math-container">$w$</span>: weight</li>
  <li><span class="math-container">$b$</span>: bias</li>
  </ul>
  <p>For example, a single layer of ANN with a linear activation function is basically a linear regression: <span class="math-container">$$ y=xw+b $$</span></p>
  <p>Using <a href="https://en.m.wikipedia.org/wiki/Hyperoperation" rel="nofollow noreferrer">hyperoperator</a>, we can construct the equivalent formula.
  <span class="math-container">$$ y=(x[2]w)[1]b$$</span></p>
  <p>Therefore, a single-unit neuron based on a hyperoperator can be defined as: <span class="math-container">$$ y=x[k]w $$</span></p>
  <p>Where <span class="math-container">$k$</span> as an operator selector, <span class="math-container">$x,w,y,k$</span> having the range of <span class="math-container">$[0,2^n-1]$</span> where <span class="math-container">$n$</span> is the number of bits, and they are all positive integer. <span class="math-container">$k$</span> and <span class="math-container">$w$</span> are parameters that can be tuned by the optimizer.</p>
  <p>It seems the model has linear activation. However, my intuition said it's possible a single neuron has non-linear activation since <span class="math-container">$k$</span> can be a parameter that is tuned by the optimizer, especially for <span class="math-container">$k\geq3$</span>, which is an hyper exponential operator. Thus, it has similarity with <a href="https://arxiv.org/abs/2404.19756" rel="nofollow noreferrer">KAN</a>, in which the activation is the tunable part</p>
  <p>Since they are all positive integers, any continuous floating point can be transformed into integer space. E.g., a normalized <span class="math-container">$P$</span> like:
  <span class="math-container">$$
  P:[0,1], P \in \mathbb{R}
  $$</span>
  is analogous to:
  <span class="math-container">$$
  P:[0,2^n-1], P \in \mathbb{N_0}
  $$</span>
  Therefore, it's just scaling and normalization issue when the inputs, outputs, and the weights in real domain.</p>
  <p>Actually, it's up to the architect how to define a single layer, like these examples, note we can replace <span class="math-container">$k$</span> with indexed <span class="math-container">$w$</span> as they are tunnable parameters:</p>
  <ol>
  <li><span class="math-container">$y=w_0[w_1]w_2$</span></li>
  <li><span class="math-container">$y=w_0[w_1]x$</span></li>
  <li><span class="math-container">$y=w_0[x]w_1$</span></li>
  <li><span class="math-container">$y=w_0[x]x$</span></li>
  <li><span class="math-container">$y=x[w_0]w_1$</span></li>
  <li><span class="math-container">$y=x[w_0]x$</span></li>
  <li><span class="math-container">$y=x[x]w_0$</span></li>
  <li><span class="math-container">$y=x[x]x$</span> (deterministic output)</li>
  </ol>
  <p>Hence, to construct a multi-layer neural network, just perform recursivity like these examples:</p>
  <ol>
  <li><span class="math-container">$y=x[k](...)$</span></li>
  <li><span class="math-container">$y=(...)[k]x$</span></li>
  <li><span class="math-container">$y=x[...](...)$</span></li>
  <li><span class="math-container">$y=x[w_0[k]w_1]x$</span></li>
  <li><span class="math-container">$y=x[x[k]x]x$</span></li>
  <li><span class="math-container">$y=x[x](...)$</span></li>
  <li><span class="math-container">$y=(...)[...](...)$</span> and so on...</li>
  </ol>
  <p>So, is hyperoperator a generalization of neural network?</p>
  
Tags: <neural-networks><activation-functions><artificial-neuron>
Title: Is hyperoperator a generalization of neural network?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Is hyperoperator a generalization of neural network?
  
  (Asked by: username_0 on 2024-12-28)
  
  We know that artificial neural network (ANN) components are:
  
  * $x$: input
  * $y$: output
  * $w$: weight
  * $b$: bias
  
  For example, a single layer of ANN with a linear activation function is basically a linear regression: $$ y=xw+b $$
  
  Using [hyperoperator](https://en.m.wikipedia.org/wiki/Hyperoperation), we can construct the equivalent formula.
  $$ y=(x[2]w)[1]b$$
  
  Therefore, a single-unit neuron based on a hyperoperator can be defined as: $$ y=x[k]w $$
  
  Where $k$ as an operator selector, $x,w,y,k$ having the range of $[0,2^n-1]$ where $n$ is the number of bits, and they are all positive integer. $k$ and $w$ are parameters that can be tuned by the optimizer.
  
  It seems the model has linear activation. However, my intuition said it's possible a single neuron has non-linear activation since $k$ can be a parameter that is tuned by the optimizer, especially for $k\geq3$, which is an hyper exponential operator. Thus, it has similarity with [KAN](https://arxiv.org/abs/2404.19756), in which the activation is the tunable part
  
  Since they are all positive integers, any continuous floating point can be transformed into integer space. E.g., a normalized $P$ like:
  $$
  P:[0,1], P \in \mathbb{R}
  $$
  is analogous to:
  $$
  P:[0,2^n-1], P \in \mathbb{N\_0}
  $$
  Therefore, it's just scaling and normalization issue when the inputs, outputs, and the weights in real domain.
  
  Actually, it's up to the architect how to define a single layer, like these examples, note we can replace $k$ with indexed $w$ as they are tunnable parameters:
  
  1. $y=w\_0[w\_1]w\_2$
  2. $y=w\_0[w\_1]x$
  3. $y=w\_0[x]w\_1$
  4. $y=w\_0[x]x$
  5. $y=x[w\_0]w\_1$
  6. $y=x[w\_0]x$
  7. $y=x[x]w\_0$
  8. $y=x[x]x$ (deterministic output)
  
  Hence, to construct a multi-layer neural network, just perform recursivity like these examples:
  
  1. $y=x[k](...)$
  2. $y=(...)[k]x$
  3. $y=x[...](...)$
  4. $y=x[w\_0[k]w\_1]x$
  5. $y=x[x[k]x]x$
  6. $y=x[x](...)$
  7. $y=(...)[...](...)$ and so on...
  
  So, is hyperoperator a generalization of neural network?
  
  **Tags:** neural-networks, activation-functions, artificial-neuron
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-29. Score: 2
  
  Perhaps to qualify as an appropriate generalization of the popular and powerful MLPs, such new types of neural networks need to be able to do stable first order gradient descent optimization which is scalable with big training dataset.
  
  However, since your proposed operator selector $k$ is a discrete parameter, optimizing over $k$ introduces challenges in training, for example, gradient-based methods struggle with discrete optimization. In addition, the interpretability of such networks becomes harder as $k$ grows, and large $k$ will obviously introduce numerical instability which impacts convergence of your proposed neural network's training.
  
  Finally hyperoperators sequence allowed Goodstein to rigorously express and study numerical growth beyond exponentiation (similar to the Ackermann function), and [Goodstein's theorem](https://en.wikipedia.org/wiki/Goodstein%27s_theorem) ultimately highlighted the incompleteness limitation of Peano Arithmetic (PA) which cannot prove that every [Goodstein sequence](https://en.wikipedia.org/wiki/Goodstein%27s_theorem#Goodstein_sequences) eventually terminates at 0, but the same can be proved in the stronger second order arithmetic or ZFC. Therefore the hyperoperators dealing with large number growth are better suited for theoretical edge case studies, but not for the everyday applications target by common ML models. Comparatively speaking, your referenced KANs may have better application since KANs have much less neurons to achieve similar performance of MLPs due to the famous [Kolmogorov-Arnold representation theorem](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_representation_theorem) linking multivariate/univariate real-valued continuous functions.
  
  > **Comments:**
  > **username_0**: The optimizer is not only about gradient descent. It can also be possible to solve it by quantum computing for the optimization problem. Put $k$ and $w$ in superposition so that it explores possible values, and let the system evolve, finding the optimized parameter based on input-output pairs.  
  > **username_1**: I don't see quantum computing is in OP's scope at all, which is yet to become applicable or relevant for everyday ML apps in the near future. Also bear in mind quantum computing as a new and potentially extremely efficient computational technique in some cases don't change computability or incompleteness edge case.  
  > **username_0**: Same for the gradient descent, which implies the question has an agnostic optimizer method.  
  
  ---
  
  