Answers.Body: <p>&quot;At inference time&quot; means &quot;when you perform inference&quot;. If &quot;inference&quot; is a synonym for &quot;forward pass&quot; (aka &quot;forward propagation&quot;) (which is not always the case in ML), then &quot;at inference time&quot;, again, means &quot;when you perform the forward pass&quot;. &quot;<a href="https://dictionary.cambridge.org/dictionary/english/at" rel="nofollow noreferrer">At</a>&quot; is just a preposition in English and it's often associated with location or time.</p>
  <p>So, the sentence</p>
  <blockquote>
  <p>you'd be training a lot more complex models which would be potentially significantly more expensive to run <strong>at inference time</strong> on the cars</p>
  </blockquote>
  <p>can be rewritten as follows</p>
  <blockquote>
  <p>you'd be training a lot more complex models which would be potentially significantly more expensive to run <strong>when performing the forward pass (i.e computing the predictions, for example, whether the traffic light is green or red)</strong> on the cars</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T21:27:27.390Z
Answers.Text: thank you. my confusion was due to `at` meaning more "at point of time" whereas "forward propagation" means some duration. What your answer tells me is that, in the context of training, duration of one "forward propagation" is small enough to be talked about as "at point of time". Am I correct at that understanding of the answer?
Answers.UserDisplayName: Martian2020
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T21:39:52.623Z
Answers.Text: The "forward propagation" is just the algorithm that we use to compute the predictions or output of the neural network. "At inference time" means when we run this algorithm (i.e. not when we're training the neural network).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T21:48:56.617Z
Answers.Text: "(i.e. not when we're training " that comment re-ignited my confusion with the question in the video. second reason (not clearly stated in the question, I see that) is the person compares "training models" (on supercomputer, as far as I recall) with doing it "at inference time on the cars". You write second part on the cars is not training. Why does he compare costs in that case?
Answers.UserDisplayName: Martian2020
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T22:17:47.020Z
Answers.Text: @Martian2020 "at inference time on the cars" means when you're using the **trained** models to perform predictions about the traffic situation or whatever.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T22:22:50.723Z
Answers.Text: I've understood that already. The person: "would be potentially significantly more expensive to run". Please explain clearly. 1. What would be and 2. more expensive to what? P.S. as of now I see he compares computing cost of training to computing cost of making prediction. What is the point in that?
Answers.UserDisplayName: Martian2020
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T22:24:46.010Z
Answers.Text: What is the point to compare costs if one cannot train while also making a prediction?
Answers.UserDisplayName: Martian2020
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T22:52:19.057Z
Answers.Text: Ok. I see now where your other doubts lie. They are talking about training multiple models and then they say that this would imply that it's more expensive to run "at inference time". The only explanation that I have is that having multiple models not only makes training more expensive but also making predictions (i.e. inference) more expensive (once the models have been trained, because we need to perform multiple forward passes: one for each model?). This is just a guess of what they could have meant. I didn't watch that video.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T23:06:15.960Z
Answers.Text: Now that you explained it clearly, I have hard time recalling the way I understood it first time I've watched it. Probably my English is not that good (and knowledge of machine learning ;-), that is why I misunderstood complex long sentence. Thanks again!       P.S. in the question the link is to exact time of the sentence.
Answers.UserDisplayName: Martian2020
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T12:45:33.413Z
Answers.LastActivityDate: 2021-12-11T12:45:33.413Z
Answers.UserDisplayName: nbro
Body: <p>I've watched <a href="https://youtu.be/j0z4FweCy4M?t=8954" rel="nofollow noreferrer">Tesla AI Day 2021</a> and there was a question Tesla staff tried to answer, but I did not quite understand the question (Note: quote taken from autogenerated subtitles, I do not hear differently, but may you will):</p>
  <blockquote>
  <p>or you'd be training a lot more complex models which would be
  potentially significantly more expensive to run at inference time on
  the cars</p>
  </blockquote>
  <p>I've found a definition of &quot;inference time&quot; in <a href="https://www.thinkautonomous.ai/blog/?p=deep-learning-optimization" rel="nofollow noreferrer">How to Optimize a Deep Learning Model for faster Inference?</a></p>
  <blockquote>
  <p>The inference time is how long is takes for a forward propagation</p>
  </blockquote>
  <p>But what does &quot;<strong>AT</strong> inference time on the cars&quot; mean? Is it just badly worded, or does this &quot;at&quot; actually add proper meaning? Also, does it make sense to run training models on the cars themselves and what can that phrase mean? Overall I do not make sense of the question. Do you?</p>
  <p>Note: I'm not a native English speaker.</p>
  
Tags: <machine-learning><terminology>
Title: What does "at inference time" on Tesla's cars mean?
UserDisplayName: Martian2020
ThreadText: # What does "at inference time" on Tesla's cars mean?
  
  (Asked by: username_0 on 2021-12-10)
  
  I've watched [Tesla AI Day 2021](https://youtu.be/j0z4FweCy4M?t=8954) and there was a question Tesla staff tried to answer, but I did not quite understand the question (Note: quote taken from autogenerated subtitles, I do not hear differently, but may you will):
  
  > or you'd be training a lot more complex models which would be
  > potentially significantly more expensive to run at inference time on
  > the cars
  
  I've found a definition of "inference time" in [How to Optimize a Deep Learning Model for faster Inference?](https://www.thinkautonomous.ai/blog/?p=deep-learning-optimization)
  
  > The inference time is how long is takes for a forward propagation
  
  But what does "**AT** inference time on the cars" mean? Is it just badly worded, or does this "at" actually add proper meaning? Also, does it make sense to run training models on the cars themselves and what can that phrase mean? Overall I do not make sense of the question. Do you?
  
  Note: I'm not a native English speaker.
  
  **Tags:** machine-learning, terminology
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-11. Score: 2 | Accepted answer
  
  "At inference time" means "when you perform inference". If "inference" is a synonym for "forward pass" (aka "forward propagation") (which is not always the case in ML), then "at inference time", again, means "when you perform the forward pass". "[At](https://dictionary.cambridge.org/dictionary/english/at)" is just a preposition in English and it's often associated with location or time.
  
  So, the sentence
  
  > you'd be training a lot more complex models which would be potentially significantly more expensive to run **at inference time** on the cars
  
  can be rewritten as follows
  
  > you'd be training a lot more complex models which would be potentially significantly more expensive to run **when performing the forward pass (i.e computing the predictions, for example, whether the traffic light is green or red)** on the cars
  
  > **Comments:**
  > **username_0**: thank you. my confusion was due to `at` meaning more "at point of time" whereas "forward propagation" means some duration. What your answer tells me is that, in the context of training, duration of one "forward propagation" is small enough to be talked about as "at point of time". Am I correct at that understanding of the answer?  
  > **username_1**: The "forward propagation" is just the algorithm that we use to compute the predictions or output of the neural network. "At inference time" means when we run this algorithm (i.e. not when we're training the neural network).  
  > **username_0**: "(i.e. not when we're training " that comment re-ignited my confusion with the question in the video. second reason (not clearly stated in the question, I see that) is the person compares "training models" (on supercomputer, as far as I recall) with doing it "at inference time on the cars". You write second part on the cars is not training. Why does he compare costs in that case?  
  > **username_1**: @username_0 "at inference time on the cars" means when you're using the **trained** models to perform predictions about the traffic situation or whatever.  
  > **username_0**: I've understood that already. The person: "would be potentially significantly more expensive to run". Please explain clearly. 1. What would be and 2. more expensive to what? P.S. as of now I see he compares computing cost of training to computing cost of making prediction. What is the point in that?  
  > **username_0**: What is the point to compare costs if one cannot train while also making a prediction?  
  > **username_1**: Ok. I see now where your other doubts lie. They are talking about training multiple models and then they say that this would imply that it's more expensive to run "at inference time". The only explanation that I have is that having multiple models not only makes training more expensive but also making predictions (i.e. inference) more expensive (once the models have been trained, because we need to perform multiple forward passes: one for each model?). This is just a guess of what they could have meant. I didn't watch that video.  
  > **username_0**: Now that you explained it clearly, I have hard time recalling the way I understood it first time I've watched it. Probably my English is not that good (and knowledge of machine learning ;-), that is why I misunderstood complex long sentence. Thanks again!       P.S. in the question the link is to exact time of the sentence.  
  
  ---
  
  