Answers.Body: <p>If you're interested in research on using LLMs for arithmetic tasks, <a href="https://arxiv.org/abs/2305.14201" rel="noreferrer">Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks</a> is a nice recent (May 2023) article.</p>
  <p>If you're interested in research on using LLMs for more advanced mathematics, <a href="https://paperswithcode.com/sota/automated-theorem-proving-on-minif2f-test" rel="noreferrer">MiniF2F</a> is a common benchmark. Check out the papers linked from the leaderboard.</p>
  <p>If you're looking for LLM models or datasets for mathematics, <a href="https://huggingface.co/hoskinson-center" rel="noreferrer">https://huggingface.co/hoskinson-center</a> and <a href="https://leandojo.org/" rel="noreferrer">https://leandojo.org/</a> are good starting points.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T14:05:34.090Z
Answers.LastActivityDate: 2023-08-11T14:05:34.090Z
Answers.UserDisplayName: Daniel Darabos
Answers.Body: <p>Depends what you mean by 'better' (in what sense?).</p>
  <p>Nevertheless, though it is possible that an LLM can solve algebra, in general it shall not offer any benefit than a computer algebra system (again depends on what is 'better', and assuming a fair comparison).</p>
  <p>This is because in doing so an LLM needs to deal with both the <em>language</em> and the <em>math</em>, while computer algebra system only needs to work on the math. It is like a top human runner and a vehicle - in general vehicle moves faster because this is what they are crafted solely for.</p>
  <p>And a few words on the comments - finding rhymes is <em>NOT</em> like solving equations, because no rhymes is <em>wrong</em> (even if it sounds terrible, but that's subjective); but there is straight <em>wrong</em> in solving equations.</p>
  <p>On the other hand, there are efforts to discover and improve algorithms, most notably <a href="https://www.deepmind.com/blog/discovering-novel-algorithms-with-alphatensor" rel="nofollow noreferrer">Google's AlphaTensor</a>. However it has nothing to do with LLM, but a lot to do with RL.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T07:13:53.140Z
Answers.Text: One can define good and incorrect rhymes, this is not subjective, this is measurable. English poetry rarely has perfect rhymes, but in Russian poetry it is quite common for poems to only have absolutely perfect rhymes. LLMs cannot make Russian poetry yet, though. When they will be able, it would be a sensation.
Answers.UserDisplayName: Anixx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T07:17:54.897Z
Answers.Text: Well, you could have specified it. And make sure this is true for all existing languages.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T07:25:08.057Z
Answers.Text: Besides, as a Go player myself, I am not arguing about whether a computer can perform better than human; we are talking about whether LLM can solve algebra better than a machine specially crafted for solving algebra, under fair assumptions.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T11:05:27.763Z
Answers.Text: But people take integrals better than CAS can.
Answers.UserDisplayName: Anixx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T01:57:23.997Z
Answers.Text: 1. Maybe, but we are comparing LLM and CAS, not CAS and human. 2. Once again, please formally define what 'better' is. 3. Anyway my statement still holds: given fair assumptions (same computing resources, same level of development effort etc.) a machine specially crafted for 1 task usually performs better than multitasking ones.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T07:02:50.037Z
Answers.LastActivityDate: 2023-08-11T07:11:19.923Z
Answers.LastEditDate: 2023-08-11T07:11:19.923Z
Answers.UserDisplayName: lpounng
Answers.Body: <blockquote>
  <p>Is it possible that LLM will gan the ability to take integrals and simplify.
  expressions better than the computer algebra system itself?</p>
  </blockquote>
  <p>Possibly, but not in one run - i.e., if you instruct it and train it to analyse and simplify step by step, while writing this out, and it fits into the context - it is an experiment.</p>
  <p>You may also want to STILL use a GAN approach with multiple LLM working on the problem - one writing, others correcting. Which requires a multi-step approach.</p>
  <p>Otherwise: no. Math has specific elements that do not really work well with a &quot;forward only&quot; approach of a neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-11T20:31:58.250Z
Answers.LastActivityDate: 2023-08-11T20:31:58.250Z
Answers.UserDisplayName: TomTom
Body: <p>What will happen if to train an LLM on taking integrals and solving equations? The process of mathematical education can be absolutely automated by a computer algebra system because the verification is easy.</p>
  <p>Is it possible that LLM will gan the ability to take integrals and simplify expressions better than the computer algebra system itself?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T19:50:17.047Z
Comments.Text: they struggle with multiplications...
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T19:51:22.363Z
Comments.Text: _"if i train a LLM on pairs  of prime numbers and their product will it learn to factorize faster than any other algorithm?"_
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T20:06:32.350Z
Comments.Text: @AlbertoSinigaglia I think, this needs to be tested.
Comments.UserDisplayName: Anixx
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T22:41:19.887Z
Comments.Text: nope, you cannot learn to do something by strictly imitating.. at most you can generalize, but you cannot extrapolate... if you want to beat human you first need to extrapolate, and then maybe some RL
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T23:06:41.103Z
Comments.Text: @AlbertoSinigaglia the best models can compose poems better than most humans can. If they can compose poems, why not tking integrals?
Comments.UserDisplayName: Anixx
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T23:20:49.640Z
Comments.Text: _best models can compose poems better than most humans_ I don't see how a subjective task can have anything to do with math
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-10T23:23:44.593Z
Comments.Text: @AlbertoSinigaglia finding rhymes is like solving equations
Comments.UserDisplayName: Anixx
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-11T08:42:43.127Z
Comments.Text: Please see this related question: https://ai.stackexchange.com/questions/25210/can-we-use-ml-to-do-anything-else-other-than-predicting-in-the-case-of-mathemat/25240#25240
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-14T02:05:26.900Z
Comments.Text: @RobinvanHoorn given all conversations in the 2 threads, I'd say OP wants to be proven right, not seeking answer nor objection.
Comments.UserDisplayName: lpounng
Tags: <math><large-language-models>
Title: What will happen if to train an LLM on mathematical exersises?
UserDisplayName: Anixx
ThreadText: # What will happen if to train an LLM on mathematical exersises?
  
  (Asked by: username_0 on 2023-08-10)
  
  What will happen if to train an LLM on taking integrals and solving equations? The process of mathematical education can be absolutely automated by a computer algebra system because the verification is easy.
  
  Is it possible that LLM will gan the ability to take integrals and simplify expressions better than the computer algebra system itself?
  
  **Tags:** math, large-language-models
  > **Comments:**
  > **username_1**: they struggle with multiplications...  
  > **username_1**: _"if i train a LLM on pairs  of prime numbers and their product will it learn to factorize faster than any other algorithm?"_  
  > **username_0**: @AlbertoSinigaglia I think, this needs to be tested.  
  > **username_1**: nope, you cannot learn to do something by strictly imitating.. at most you can generalize, but you cannot extrapolate... if you want to beat human you first need to extrapolate, and then maybe some RL  
  > **username_0**: @AlbertoSinigaglia the best models can compose poems better than most humans can. If they can compose poems, why not tking integrals?  
  > **username_1**: _best models can compose poems better than most humans_ I don't see how a subjective task can have anything to do with math  
  > **username_0**: @AlbertoSinigaglia finding rhymes is like solving equations  
  > **username_2**: Please see this related question: https://ai.stackexchange.com/questions/25210/can-we-use-ml-to-do-anything-else-other-than-predicting-in-the-case-of-mathemat/25240#25240  
  > **username_3**: @RobinvanHoorn given all conversations in the 2 threads, I'd say OP wants to be proven right, not seeking answer nor objection.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2023-08-11. Score: 5
  
  If you're interested in research on using LLMs for arithmetic tasks, [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://arxiv.org/abs/2305.14201) is a nice recent (May 2023) article.
  
  If you're interested in research on using LLMs for more advanced mathematics, [MiniF2F](https://paperswithcode.com/sota/automated-theorem-proving-on-minif2f-test) is a common benchmark. Check out the papers linked from the leaderboard.
  
  If you're looking for LLM models or datasets for mathematics, <https://huggingface.co/hoskinson-center> and <https://leandojo.org/> are good starting points.
  
  
  ---
  
  ### Answer by username_3 on 2023-08-11. Score: 3
  
  Depends what you mean by 'better' (in what sense?).
  
  Nevertheless, though it is possible that an LLM can solve algebra, in general it shall not offer any benefit than a computer algebra system (again depends on what is 'better', and assuming a fair comparison).
  
  This is because in doing so an LLM needs to deal with both the *language* and the *math*, while computer algebra system only needs to work on the math. It is like a top human runner and a vehicle - in general vehicle moves faster because this is what they are crafted solely for.
  
  And a few words on the comments - finding rhymes is *NOT* like solving equations, because no rhymes is *wrong* (even if it sounds terrible, but that's subjective); but there is straight *wrong* in solving equations.
  
  On the other hand, there are efforts to discover and improve algorithms, most notably [Google's AlphaTensor](https://www.deepmind.com/blog/discovering-novel-algorithms-with-alphatensor). However it has nothing to do with LLM, but a lot to do with RL.
  
  > **Comments:**
  > **username_0**: One can define good and incorrect rhymes, this is not subjective, this is measurable. English poetry rarely has perfect rhymes, but in Russian poetry it is quite common for poems to only have absolutely perfect rhymes. LLMs cannot make Russian poetry yet, though. When they will be able, it would be a sensation.  
  > **username_3**: Well, you could have specified it. And make sure this is true for all existing languages.  
  > **username_3**: Besides, as a Go player myself, I am not arguing about whether a computer can perform better than human; we are talking about whether LLM can solve algebra better than a machine specially crafted for solving algebra, under fair assumptions.  
  > **username_0**: But people take integrals better than CAS can.  
  > **username_3**: 1. Maybe, but we are comparing LLM and CAS, not CAS and human. 2. Once again, please formally define what 'better' is. 3. Anyway my statement still holds: given fair assumptions (same computing resources, same level of development effort etc.) a machine specially crafted for 1 task usually performs better than multitasking ones.  
  
  ---
  
  ### Answer by username_5 on 2023-08-11. Score: 1
  
  > Is it possible that LLM will gan the ability to take integrals and simplify.
  > expressions better than the computer algebra system itself?
  
  Possibly, but not in one run - i.e., if you instruct it and train it to analyse and simplify step by step, while writing this out, and it fits into the context - it is an experiment.
  
  You may also want to STILL use a GAN approach with multiple LLM working on the problem - one writing, others correcting. Which requires a multi-step approach.
  
  Otherwise: no. Math has specific elements that do not really work well with a "forward only" approach of a neural network.
  
  
  ---
  
  