Answers.Body: <p>Given the context and <a href="https://www.google.com/search?q=inference%20kernel%20site%3Aopenai.com" rel="nofollow noreferrer">similar usage</a>, they're probably referring to compute kernels in GPUs. <a href="https://openai.com/research/triton" rel="nofollow noreferrer">This OpenAI blog post</a> seems informative for more on this topic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-29T22:13:50.917Z
Answers.LastActivityDate: 2024-02-29T22:13:50.917Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>I spoke briefly with ChatGPT on the matter; 3.5, I don't have access to 4.0. It says that it's not anything that it has or uses, so this is post-2022, so it's not anything that has to do with prompt replies in generic GPT systems, though it could have something to do with 4.0-related enhancements in this area.</p>
  <p>As for &quot;Inference Kernel&quot;, for LCF-systems (as it reminded me), you're essentially talking about <a href="https://en.wikipedia.org/wiki/Logic_for_Computable_Functions" rel="nofollow noreferrer">Milner's type-checking system</a>. I don't think 4.0 is implementing a Milner-type theorem prover or proof assistant as an &quot;Inference Kernel&quot;, and 3.5 also expresses skepticism that this is what the term is referring to.</p>
  <p>The use of &quot;ML&quot; in the <a href="https://openai.com/careers/hwsw-co-design-engineer" rel="nofollow noreferrer">Open AI HW/SW Co-Design Engineer Job Ad</a> that was alluded to, indirectly, by the previous reply, alongside &quot;GPU&quot; and &quot;CUDA&quot;, is almost certainly &quot;Machine Learning&quot;, rather than Milner's LCF-based <a href="https://en.wikipedia.org/wiki/ML_(programming_language)" rel="nofollow noreferrer">ML language</a>. It would be interesting if their inference kernel did, indeed, have connection to this, since Milner's LCF and ML have evolved into <a href="https://en.wikipedia.org/wiki/HOL_(proof_assistant)" rel="nofollow noreferrer">Higher Order Logic (HOL)</a> proof assistants, and there actually are such things as <a href="https://www.degruyter.com/document/doi/10.1515/itit-2019-0001/html?lang=en" rel="nofollow noreferrer">Higher Order Logic Theorem <strong>Provers</strong></a> (like this <a href="https://hol-theorem-prover.org/" rel="nofollow noreferrer">HOL Theorem Prover</a> dot-com); which is an entire echelon above <a href="https://en.wikipedia.org/wiki/Automated_theorem_proving" rel="nofollow noreferrer">old-school theorem provers</a>, and a step beyond <a href="https://en.wikipedia.org/wiki/Proof_assistant" rel="nofollow noreferrer">proof assistants</a>. The challenge of hybridizing connectionist / neural net &amp; non-linear statistical models with old-school symbolic / logic-based models is still an open area of research.</p>
  <p>Since the discussion is in the context of GPU configurations, 3.5 lists &quot;hardware compatibility&quot;, &quot;software bugs&quot;, and &quot;optimization issues&quot; that particularly apply to the GPU implementation of the inference kernel as issues to examine, and it suggests it's an issue that has to do with the GPU-based implementation of the &quot;inference kernel&quot;. The link (ultimately) alluded to in the previous reply to <a href="https://github.com/openai/triton" rel="nofollow noreferrer">Triton (GitHub)</a> may be what this has to do with.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-02T09:14:07.247Z
Answers.LastActivityDate: 2024-05-02T09:20:58.667Z
Answers.LastEditDate: 2024-05-02T09:20:58.667Z
Answers.UserDisplayName: NinjaDarth
Body: <p>In the recent event where ChatGPT &quot;went crazy&quot;, this term was used in the official post-mortem to describe what happened:</p>
  <blockquote>
  <p>In this case, the bug was in the step where the model chooses these numbers. Akin to being lost in translation, the model chose slightly wrong numbers, which produced word sequences that made no sense. More technically, <strong>inference kernels</strong> produced incorrect results when used in certain GPU configurations.</p>
  </blockquote>
  <p>Now I know what <strong>inference</strong> is, at least wrt LLMs. It's when you use a trained LLM to generate text by giving it a prompt (feel free to offer a more technical clarification if necessary). As for <strong>kernel</strong> that has several meanings in computing generally, such as an <a href="https://en.wikipedia.org/wiki/Kernel_(operating_system)" rel="nofollow noreferrer">operating system kernel</a> being the core part of the <strong>OS</strong>, often in contrast to shells, including command lines and GUIs. Then there is the <a href="https://en.wikipedia.org/wiki/Kernel_(image_processing)" rel="nofollow noreferrer">sense of kernel in image processing</a>, also known as a <strong>convolution matrix</strong>. There's also <strong>Compute Kernels</strong> in <strong>GPGPU</strong> programming, with which I am the least familiar.</p>
  <p>But what does the overall phrase &quot;inference kernel&quot; refer to technically? My hunch is that it's some kind of specialization of the GPU sense, though I read <a href="https://www.lesswrong.com/posts/koNsuop5AHxBG5D7t/can-someone-explain-to-me-what-went-wrong-with-chatgpt?commentId=5ts6boALcsvFHS9Ba" rel="nofollow noreferrer">a comment in an AI-adjacent forum</a> that it's related to the OS sense, which I doubt is accurate.</p>
  <p>Other than that I can't actually find this term mentioned very much, and I can't find it defined at all.</p>
  <hr />
  <p><strong>EDIT</strong></p>
  <p><a href="http://www.proof-technologies.com/misc/glossary.html" rel="nofollow noreferrer">I found a definition of <em>inference kernel</em></a> but it's about automatic theorom provers and doesn't seem to be applicable to AI, not that I can follow it very well:</p>
  <blockquote>
  <p><strong>inference kernel</strong> : (n) The part of an LCF-style system that implements the deductive system of its formal logic, and relied upon by all subsequent deductive functionality for its soundness. It defines the datatype for internal theorems as an abstract datatype, with the primitive inference rules and the primitive assertion commands as its only primitive constructors. The inference kernel is a crucial trusted component of its system. See &quot;language kernel&quot;, &quot;logical core&quot;, &quot;core system&quot;.</p>
  </blockquote>
  
Tags: <terminology><large-language-models><chatgpt><inference>
Title: What is an "inference kernel"?
UserDisplayName: hippietrail
ThreadText: # What is an "inference kernel"?
  
  (Asked by: username_0 on 2024-02-26)
  
  In the recent event where ChatGPT "went crazy", this term was used in the official post-mortem to describe what happened:
  
  > In this case, the bug was in the step where the model chooses these numbers. Akin to being lost in translation, the model chose slightly wrong numbers, which produced word sequences that made no sense. More technically, **inference kernels** produced incorrect results when used in certain GPU configurations.
  
  Now I know what **inference** is, at least wrt LLMs. It's when you use a trained LLM to generate text by giving it a prompt (feel free to offer a more technical clarification if necessary). As for **kernel** that has several meanings in computing generally, such as an [operating system kernel](https://en.wikipedia.org/wiki/Kernel_(operating_system)) being the core part of the **OS**, often in contrast to shells, including command lines and GUIs. Then there is the [sense of kernel in image processing](https://en.wikipedia.org/wiki/Kernel_(image_processing)), also known as a **convolution matrix**. There's also **Compute Kernels** in **GPGPU** programming, with which I am the least familiar.
  
  But what does the overall phrase "inference kernel" refer to technically? My hunch is that it's some kind of specialization of the GPU sense, though I read [a comment in an AI-adjacent forum](https://www.lesswrong.com/posts/koNsuop5AHxBG5D7t/can-someone-explain-to-me-what-went-wrong-with-chatgpt?commentId=5ts6boALcsvFHS9Ba) that it's related to the OS sense, which I doubt is accurate.
  
  Other than that I can't actually find this term mentioned very much, and I can't find it defined at all.
  
  ---
  
  **EDIT**
  
  [I found a definition of *inference kernel*](http://www.proof-technologies.com/misc/glossary.html) but it's about automatic theorom provers and doesn't seem to be applicable to AI, not that I can follow it very well:
  
  > **inference kernel** : (n) The part of an LCF-style system that implements the deductive system of its formal logic, and relied upon by all subsequent deductive functionality for its soundness. It defines the datatype for internal theorems as an abstract datatype, with the primitive inference rules and the primitive assertion commands as its only primitive constructors. The inference kernel is a crucial trusted component of its system. See "language kernel", "logical core", "core system".
  
  **Tags:** terminology, large-language-models, chatgpt, inference
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-02-29. Score: 1
  
  Given the context and [similar usage](https://www.google.com/search?q=inference%20kernel%20site%3Aopenai.com), they're probably referring to compute kernels in GPUs. [This OpenAI blog post](https://openai.com/research/triton) seems informative for more on this topic.
  
  
  ---
  
  ### Answer by username_2 on 2024-05-02. Score: 0
  
  I spoke briefly with ChatGPT on the matter; 3.5, I don't have access to 4.0. It says that it's not anything that it has or uses, so this is post-2022, so it's not anything that has to do with prompt replies in generic GPT systems, though it could have something to do with 4.0-related enhancements in this area.
  
  As for "Inference Kernel", for LCF-systems (as it reminded me), you're essentially talking about [Milner's type-checking system](https://en.wikipedia.org/wiki/Logic_for_Computable_Functions). I don't think 4.0 is implementing a Milner-type theorem prover or proof assistant as an "Inference Kernel", and 3.5 also expresses skepticism that this is what the term is referring to.
  
  The use of "ML" in the [Open AI HW/SW Co-Design Engineer Job Ad](https://openai.com/careers/hwsw-co-design-engineer) that was alluded to, indirectly, by the previous reply, alongside "GPU" and "CUDA", is almost certainly "Machine Learning", rather than Milner's LCF-based [ML language](https://en.wikipedia.org/wiki/ML_(programming_language)). It would be interesting if their inference kernel did, indeed, have connection to this, since Milner's LCF and ML have evolved into [Higher Order Logic (HOL)](https://en.wikipedia.org/wiki/HOL_(proof_assistant)) proof assistants, and there actually are such things as [Higher Order Logic Theorem **Provers**](https://www.degruyter.com/document/doi/10.1515/itit-2019-0001/html?lang=en) (like this [HOL Theorem Prover](https://hol-theorem-prover.org/) dot-com); which is an entire echelon above [old-school theorem provers](https://en.wikipedia.org/wiki/Automated_theorem_proving), and a step beyond [proof assistants](https://en.wikipedia.org/wiki/Proof_assistant). The challenge of hybridizing connectionist / neural net & non-linear statistical models with old-school symbolic / logic-based models is still an open area of research.
  
  Since the discussion is in the context of GPU configurations, 3.5 lists "hardware compatibility", "software bugs", and "optimization issues" that particularly apply to the GPU implementation of the inference kernel as issues to examine, and it suggests it's an issue that has to do with the GPU-based implementation of the "inference kernel". The link (ultimately) alluded to in the previous reply to [Triton (GitHub)](https://github.com/openai/triton) may be what this has to do with.
  
  
  ---
  
  