Answers.Body: <p>Retrofitting (as you might intuit) is the hardest part.  I've been working toward a solution.  <code>Fraction</code> (from <code>fractions</code>) plugs and plays quite nicely with <code>floats</code> in many areas.  I was surprised how smoothly <code>numpy</code> integrates matrix functionality as if you had used <code>float</code>.  The first hurdle I found was <code>sqrt()</code>, which just fails.  So if you have an <code>ndarray</code> composed of <code>Fractions</code>, for simple math you get <code>ndarrays</code> composed of <code>Fractions</code>.  But if, for instance, you have a vector of 5 <code>Fractions</code>, <code>np.std()</code> just fails, because of the <code>sqrt()</code> issue.  But <code>np.var()</code> returns a <code>Fraction</code>.
  My current solution is to create a <em><strong>class</strong></em> that inherits from <code>Fraction</code> and make sure that <em><strong>class</strong></em> has a <code>sqrt()</code> method.  To my elation, passing a vector of those <em><strong>objects</strong></em> to <code>np.std()</code> will return a rational answer of that <em><strong>object</strong></em> type.</p>
  <p>As this is a work in progress, I have not tried this with Keras, TF, PyTorch, etc, yet.  I highly suspect that there will be myriad hurdles, there!</p>
  <p>And the best part of <code>Fraction</code> (from <code>fractions</code>) is that both the numerator and the denominator are python <code>integers</code>, so they easily scale to any size, like GMP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T20:30:52.567Z
Answers.LastActivityDate: 2024-11-29T23:38:29.870Z
Answers.LastEditDate: 2024-11-29T23:38:29.870Z
Answers.UserDisplayName: James Huddle
Body: <p>Rational numbers would help alleviate some gradient issues by not losing precision as the weights and the propagated values (signal) reach extremely low and high values.</p>
  <p>I'm not aware of any hardware that is optimized for rationals. GPUs are all optimized for vector/matrix operations on floats. So, a couple of negatives are that it would likely double the storage and computation necessary for an ANN. But meh, hardware is constantly improving, and if there was value in creating a rational co-processor for AI, someone would build it.</p>
  <p>I suspect an advantage would be greater precision and possibly mitigating some aspects of gradient issues.</p>
  <p>What are the other pros and cons of an ANN using rationals?</p>
  <p>Are there any research papers investigating the use of rationals in ANNs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-27T18:58:53.383Z
Comments.Text: I just discovered a multi-precision library that supports GPU parallel calculations on rationals.  There is a port of GMP (GNU Multi Precision Library) to Cuda called CUMP. https://github.com/skystar0227/CUMP
Comments.UserDisplayName: Mark_Sagecy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-27T19:07:56.423Z
Comments.Text: My search for rational neural networks found research using "rational activation functions."  In other words, each neuron has a numerator function and a denominator function for activation. This does not imply the values in the network are stored or propagated as rationals.  However, I think the two concepts fit together nicely. https://arxiv.org/abs/2004.01902
Comments.UserDisplayName: Mark_Sagecy
Tags: <weights><gradient><vanishing-gradient-problem><precision>
Title: Might use of rational numbers and calculations be beneficial for an ANN?
UserDisplayName: Mark_Sagecy
ThreadText: # Might use of rational numbers and calculations be beneficial for an ANN?
  
  (Asked by: username_0 on 2024-03-27)
  
  Rational numbers would help alleviate some gradient issues by not losing precision as the weights and the propagated values (signal) reach extremely low and high values.
  
  I'm not aware of any hardware that is optimized for rationals. GPUs are all optimized for vector/matrix operations on floats. So, a couple of negatives are that it would likely double the storage and computation necessary for an ANN. But meh, hardware is constantly improving, and if there was value in creating a rational co-processor for AI, someone would build it.
  
  I suspect an advantage would be greater precision and possibly mitigating some aspects of gradient issues.
  
  What are the other pros and cons of an ANN using rationals?
  
  Are there any research papers investigating the use of rationals in ANNs?
  
  **Tags:** weights, gradient, vanishing-gradient-problem, precision
  
  > **Comments:**
  > **username_0**: I just discovered a multi-precision library that supports GPU parallel calculations on rationals.  There is a port of GMP (GNU Multi Precision Library) to Cuda called CUMP. https://github.com/skystar0227/CUMP  
  > **username_0**: My search for rational neural networks found research using "rational activation functions."  In other words, each neuron has a numerator function and a denominator function for activation. This does not imply the values in the network are stored or propagated as rationals.  However, I think the two concepts fit together nicely. https://arxiv.org/abs/2004.01902  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-29. Score: 0
  
  Retrofitting (as you might intuit) is the hardest part. I've been working toward a solution. `Fraction` (from `fractions`) plugs and plays quite nicely with `floats` in many areas. I was surprised how smoothly `numpy` integrates matrix functionality as if you had used `float`. The first hurdle I found was `sqrt()`, which just fails. So if you have an `ndarray` composed of `Fractions`, for simple math you get `ndarrays` composed of `Fractions`. But if, for instance, you have a vector of 5 `Fractions`, `np.std()` just fails, because of the `sqrt()` issue. But `np.var()` returns a `Fraction`.
  My current solution is to create a ***class*** that inherits from `Fraction` and make sure that ***class*** has a `sqrt()` method. To my elation, passing a vector of those ***objects*** to `np.std()` will return a rational answer of that ***object*** type.
  
  As this is a work in progress, I have not tried this with Keras, TF, PyTorch, etc, yet. I highly suspect that there will be myriad hurdles, there!
  
  And the best part of `Fraction` (from `fractions`) is that both the numerator and the denominator are python `integers`, so they easily scale to any size, like GMP.
  
  
  ---
  
  