Answers.Body: <p>Due to their architecture, GPUS is faster on floating numbers rather than integers and binary numbers like one and zero. they are designed to perform floating point arithmetic operations such as matrix operations, image rending, graphical processing, and scientific operations.</p>
  <p>and the GPUs use the SIMD (Single Instruction Multiple Data) model, where a single operation runs on multiple data points. For that type of task, floating points are mostly suitable.</p>
  <p>The pipeline in a GPU is highly optimized for floating-point operations, including addition, multiplication, and division. Processing integers or zeros might not fully utilize these pipelines, resulting in under-utilization of GPU resources.</p>
  <p>While zeros are computationally less expensive in some scenarios (e.g., sparse matrices), GPUs often don't have specific optimizations for them unless explicitly handled in software. Floating-point operations are given more attention for general workloads</p>
  <p>In simple words, GPUs mostly run fast on floating numbers rather than zeros because they are designed to work on floating operations :), and working with zero will not fully utilize the gpu usage.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T02:25:16.483Z
Answers.Text: While I agree with you in general, `torch.zeros` returns the same data type as `torch.randn`. The default type is float32.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T11:19:20.470Z
Answers.Text: Actually torch doesn't even support multiplying int32 and float32 matrices together! `expected mat1 and mat2 to have the same dtype, but got: int != float`. And at least on my setup multiplying int32 matrices together isn't implemented either: `"addmm_cuda" not implemented for 'Int'`.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-02T04:15:54.060Z
Answers.Text: Yes @NikoNyrh, you are correct about the torch. zeros and torch. since both are using the float, I think the issue is due to not using torch.cuda.synchronize()
Answers.UserDisplayName: Keval Pandya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-28T06:14:43.803Z
Answers.LastActivityDate: 2024-11-28T06:14:43.803Z
Answers.UserDisplayName: Keval Pandya
Answers.Body: <p><code>torch.zeros</code> returns an array of <code>torch.float32</code>, just as <code>torch.randn</code>, so I don't think Keval's explanation is correct.</p>
  <p>Here is a simple benchmarking script:</p>
  <pre><code>import time
  import torch
  import numpy as np
  
  device = 'cuda'
  a = torch.zeros((10000,10000)).to(device)
  b = torch.randn((10000,10000)).to(device)
  c = torch.randn((10000,10000)).to(device)
  
  times = [time.time()]
  while times[-1] - times[0] &lt; 3:  # Run for 3 seconds
      tmp1 = a @ b
      times.append(time.time())
      torch.cuda.synchronize()
      times.append(time.time())
  
      tmp2 = c @ b
      times.append(time.time())
      torch.cuda.synchronize()
      times.append(time.time())
  
  
  
  times = np.array(times)
  times = (times[1:] - times[:-1]).reshape((len(times) // 4, 4))
  
  
  np.log10(times).round(2)[:10]
  '''
  array([[-2.97, -1.27, -3.78, -1.04],
         [-3.85, -1.27, -3.89, -1.25],
         [-4.  , -1.22, -3.92, -1.16],
         [-3.87, -1.09, -3.73, -1.04],
         [-3.97, -0.97, -3.75, -0.98],
         [-3.69, -1.  , -4.  , -0.97],
         [-3.85, -1.02, -3.95, -0.97],
         [-3.88, -0.99, -3.93, -0.96],
         [-3.85, -0.98, -3.69, -0.98],
         [-3.95, -0.99, -3.95, -0.97]])
  '''
  </code></pre>
  <p>Note how calling <code>a @ b</code> or <code>c @ b</code> takes barely any time, the torch API is asynchronous (we can run GPU and Python code in parallel). Thus we need to call <code>torch.cuda.synchronize()</code> to measure how long the calculation took.</p>
  <p>I actually made a big mistake on the first version of the script, omitting the synchronization call.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T02:21:11.720Z
Answers.LastActivityDate: 2024-11-30T02:34:31.943Z
Answers.LastEditDate: 2024-11-30T02:34:31.943Z
Answers.UserDisplayName: NikoNyrh
Body: <p>It seems no difference for cpu but not for gpu</p>
  <p><a href="https://i.sstatic.net/QauGZwnZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QauGZwnZ.png" alt="enter image description here" /></a> <a href="https://i.sstatic.net/Wn5GpcwX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Wn5GpcwX.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-27T10:56:51.420Z
Comments.Text: The GPU times are tiny, literally 1000s of times faster than CPU ones, so are going to be heavily influenced by random fluctuations and any side issues the Python interpreter has. I would suggest running within a proper benchmarking function which will run them multiple times to get a decent average.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-28T12:28:59.043Z
Comments.Text: I don't understand your question, *...float numbers rather than zeros?*, what do you mean by that? zeros can be represented in float or integer, but not both.
Comments.UserDisplayName: Muhammad Ikhwan Perwira
Tags: <gpu>
Title: Why gpu works faster for float number rather then zeros?
UserDisplayName: Тима 
ThreadText: # Why gpu works faster for float number rather then zeros?
  
  (Asked by: username_0 on 2024-11-27)
  
  It seems no difference for cpu but not for gpu
  
  [![enter image description here](https://i.sstatic.net/QauGZwnZ.png)](https://i.sstatic.net/QauGZwnZ.png) [![enter image description here](https://i.sstatic.net/Wn5GpcwX.png)](https://i.sstatic.net/Wn5GpcwX.png)
  
  **Tags:** gpu
  
  > **Comments:**
  > **username_1**: The GPU times are tiny, literally 1000s of times faster than CPU ones, so are going to be heavily influenced by random fluctuations and any side issues the Python interpreter has. I would suggest running within a proper benchmarking function which will run them multiple times to get a decent average.  
  > **username_2**: I don't understand your question, *...float numbers rather than zeros?*, what do you mean by that? zeros can be represented in float or integer, but not both.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-11-28. Score: 1
  
  Due to their architecture, GPUS is faster on floating numbers rather than integers and binary numbers like one and zero. they are designed to perform floating point arithmetic operations such as matrix operations, image rending, graphical processing, and scientific operations.
  
  and the GPUs use the SIMD (Single Instruction Multiple Data) model, where a single operation runs on multiple data points. For that type of task, floating points are mostly suitable.
  
  The pipeline in a GPU is highly optimized for floating-point operations, including addition, multiplication, and division. Processing integers or zeros might not fully utilize these pipelines, resulting in under-utilization of GPU resources.
  
  While zeros are computationally less expensive in some scenarios (e.g., sparse matrices), GPUs often don't have specific optimizations for them unless explicitly handled in software. Floating-point operations are given more attention for general workloads
  
  In simple words, GPUs mostly run fast on floating numbers rather than zeros because they are designed to work on floating operations :), and working with zero will not fully utilize the gpu usage.
  
  > **Comments:**
  > **username_4**: While I agree with you in general, `torch.zeros` returns the same data type as `torch.randn`. The default type is float32.  
  > **username_4**: Actually torch doesn't even support multiplying int32 and float32 matrices together! `expected mat1 and mat2 to have the same dtype, but got: int != float`. And at least on my setup multiplying int32 matrices together isn't implemented either: `"addmm_cuda" not implemented for 'Int'`.  
  > **username_3**: Yes @username_4, you are correct about the torch. zeros and torch. since both are using the float, I think the issue is due to not using torch.cuda.synchronize()  
  
  ---
  
  ### Answer by username_4 on 2024-11-30. Score: 1
  
  `torch.zeros` returns an array of `torch.float32`, just as `torch.randn`, so I don't think username_3's explanation is correct.
  
  Here is a simple benchmarking script:
  
  ```
  import time
  import torch
  import numpy as np
  
  device = 'cuda'
  a = torch.zeros((10000,10000)).to(device)
  b = torch.randn((10000,10000)).to(device)
  c = torch.randn((10000,10000)).to(device)
  
  times = [time.time()]
  while times[-1] - times[0] < 3:  # Run for 3 seconds
      tmp1 = a @ b
      times.append(time.time())
      torch.cuda.synchronize()
      times.append(time.time())
  
      tmp2 = c @ b
      times.append(time.time())
      torch.cuda.synchronize()
      times.append(time.time())
  
  times = np.array(times)
  times = (times[1:] - times[:-1]).reshape((len(times) // 4, 4))
  
  np.log10(times).round(2)[:10]
  '''
  array([[-2.97, -1.27, -3.78, -1.04],
         [-3.85, -1.27, -3.89, -1.25],
         [-4.  , -1.22, -3.92, -1.16],
         [-3.87, -1.09, -3.73, -1.04],
         [-3.97, -0.97, -3.75, -0.98],
         [-3.69, -1.  , -4.  , -0.97],
         [-3.85, -1.02, -3.95, -0.97],
         [-3.88, -0.99, -3.93, -0.96],
         [-3.85, -0.98, -3.69, -0.98],
         [-3.95, -0.99, -3.95, -0.97]])
  '''
  
  ```
  
  Note how calling `a @ b` or `c @ b` takes barely any time, the torch API is asynchronous (we can run GPU and Python code in parallel). Thus we need to call `torch.cuda.synchronize()` to measure how long the calculation took.
  
  I actually made a big mistake on the first version of the script, omitting the synchronization call.
  
  
  ---
  
  