Answers.Body: <p>Yes, your interpretation is correct, it's also stressed out in the documentation: <em>&quot;Time per inference step is the average of 30 batches and 10 repetitions.&quot;</em>.</p>
  <p>Be aware that the relation between batch size and inference time is not linear, so you can't halve of double the time reported on that table to estimate inference time for different batch size. And it's also very specific to the GPU used, so variations with different hardware must be expected.</p>
  <p>Also, the table in keras documentation refers to the first definition of inference time you linked, which is more precise but less useful for real use cases. So if timing is crucial for you be sure to time also every preprocessing and post processing step that you're applying to your images and predictions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-06T07:17:39.070Z
Answers.LastActivityDate: 2022-10-06T07:17:39.070Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I was looking at the table presented in <a href="https://keras.io/api/applications/" rel="nofollow noreferrer">Keras Applications</a> and there is a column that says <strong>Time (ms) per inference step (CPU)</strong> and another for GPU. I did a quick research on the internet and I came across some explanations of what inference time is as we can see 1-<a href="https://www.thinkautonomous.ai/blog/deep-learning-optimization/#inference" rel="nofollow noreferrer">here </a> and also 2-<a href="https://www.quora.com/What-is-inference-time-in-deep-learning" rel="nofollow noreferrer">here</a>.</p>
  <pre><code>1-The inference time is how long is takes for a forward propagation. To get the number of Frames per Second, we divide 1/inference time.
  
  
  2-In deep learning, inference time is the amount of time it takes for a machine learning model to process new data and make a prediction.
  </code></pre>
  <p>But the explanations are talking about <strong>inference time</strong> and not <strong>Time per inference step</strong>, so I was wondering if there is some difference to it? Is this time related to <em>steps_per_epoch</em> or <em>validation_steps</em> from <code>fit</code> function?</p>
  <p>They also mention these two lines in the Keras website :</p>
  <pre><code>Time per inference step is the average of 30 batches and 10 repetitions.
  Batch size: 32
  </code></pre>
  <p>Does this mean that the <strong>Time per inference step</strong> is the average time it takes the model to make predictions for each batch of 32 samples as opposed to just one sample?</p>
  <p>Thanks in advance!</p>
  
Tags: <machine-learning><deep-learning>
Title: Is there a difference between 'inference step time' and 'inference time'?
UserDisplayName: Blue Ross
ThreadText: # Is there a difference between 'inference step time' and 'inference time'?
  
  (Asked by: username_0 on 2022-10-05)
  
  I was looking at the table presented in [Keras Applications](https://keras.io/api/applications/) and there is a column that says **Time (ms) per inference step (CPU)** and another for GPU. I did a quick research on the internet and I came across some explanations of what inference time is as we can see 1-[here](https://www.thinkautonomous.ai/blog/deep-learning-optimization/#inference)  and also 2-[here](https://www.quora.com/What-is-inference-time-in-deep-learning).
  
  ```
  1-The inference time is how long is takes for a forward propagation. To get the number of Frames per Second, we divide 1/inference time.
  
  2-In deep learning, inference time is the amount of time it takes for a machine learning model to process new data and make a prediction.
  
  ```
  
  But the explanations are talking about **inference time** and not **Time per inference step**, so I was wondering if there is some difference to it? Is this time related to *steps\_per\_epoch* or *validation\_steps* from `fit` function?
  
  They also mention these two lines in the Keras website :
  
  ```
  Time per inference step is the average of 30 batches and 10 repetitions.
  Batch size: 32
  
  ```
  
  Does this mean that the **Time per inference step** is the average time it takes the model to make predictions for each batch of 32 samples as opposed to just one sample?
  
  Thanks in advance!
  
  **Tags:** machine-learning, deep-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-06. Score: 1 | Accepted answer
  
  Yes, your interpretation is correct, it's also stressed out in the documentation: *"Time per inference step is the average of 30 batches and 10 repetitions."*.
  
  Be aware that the relation between batch size and inference time is not linear, so you can't halve of double the time reported on that table to estimate inference time for different batch size. And it's also very specific to the GPU used, so variations with different hardware must be expected.
  
  Also, the table in keras documentation refers to the first definition of inference time you linked, which is more precise but less useful for real use cases. So if timing is crucial for you be sure to time also every preprocessing and post processing step that you're applying to your images and predictions.
  
  
  ---
  
  