Answers.Body: <p>memory pooling is not really much of a thing these days: the interface is not really that all of a sudden you get a single address space. You still have individual GPUs, you just specify the enablement of peer transfers. This makes sense from a design point of view, because in order to write efficient software the software has to really be aware of which data is on which physical device so that efforts can be made to optimize and reduce (unnecessary) transfers.</p>
  <p>Therefore I'd say the premise of your question is flawed. Perhaps it should be edited.</p>
  <p>Anyway, I'm running llama.cpp with dual 3090 with NVLink enabled. llama.cpp does have implemented peer transfers and they can significantly speed up inference. For example 10 tok/s -&gt; 17 tok/s for a 70B model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-11T06:37:51.677Z
Answers.Text: I get 17.1 tok/s for a 70B model on dual 3090 WITHOUT NVLink, using a llamafile. Don't have NVLink to test
Answers.UserDisplayName: MrMartin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-12T05:45:53.470Z
Answers.Text: Yes my answer is outdated, I have known for a few months now that it is indeed possible to have very little data volume shuttling back and forth to inference LLMs across two GPUs. now the GPUs aren't going to be each fully loaded, but you do leverage both their memories so you are still winning. 
  
  NVLink is useful for various training use cases. And simulations and other HPC type workloads.
Answers.UserDisplayName: Steven Lu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-14T09:39:55.690Z
Answers.Text: Can you report the speed you get with and without NVLink now, on something like [70B-LLama](https://huggingface.co/Mozilla/Meta-Llama-3-70B-Instruct-llamafile)?
Answers.UserDisplayName: MrMartin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-14T15:35:59.077Z
Answers.Text: There isn't a way to [disable nvlink](https://forums.developer.nvidia.com/t/need-example-to-disable-nvlink/70818/4) in software if it is physically connected. I could physically disconnect it but my system setup is a bit unconventional and I can't quite do that right now, sorry. But by monitoring the data transferred via some command like `watch -n.5 nvidia-smi nvlink -gt d` you can get a good sense of how much the nvlink is being utilized; when below a rate your PCIe bus can provide (which could vary by a factor of up to 64 between x1 gen 2 and x16 gen 4!) it won't provide much benefit.
Answers.UserDisplayName: Steven Lu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-30T01:38:28.077Z
Answers.LastActivityDate: 2023-09-30T01:38:28.077Z
Answers.UserDisplayName: Steven Lu
Body: <p>Apparantly its possible to pool the memory of two 3090 using NVLink (although not with 4090). This would make it possible to run large LLM's on consumer hardware.</p>
  <p><a href="https://huggingface.co/transformers/v4.9.2/performance.html" rel="nofollow noreferrer">https://huggingface.co/transformers/v4.9.2/performance.html</a></p>
  <p>Although before I invest into a new GPU, I would like to verify that it actually works, since conventional wisdom used to be that SLI only doubled performance, not memory.</p>
  <p>So has anyone tried yet? Whats the token rate?</p>
  
Tags: <gpu>
Title: Has anyone tried to use llama.cpp with NVLink?
UserDisplayName: user2741831
ThreadText: # Has anyone tried to use llama.cpp with NVLink?
  
  (Asked by: username_0 on 2023-09-14)
  
  Apparantly its possible to pool the memory of two 3090 using NVLink (although not with 4090). This would make it possible to run large LLM's on consumer hardware.
  
  <https://huggingface.co/transformers/v4.9.2/performance.html>
  
  Although before I invest into a new GPU, I would like to verify that it actually works, since conventional wisdom used to be that SLI only doubled performance, not memory.
  
  So has anyone tried yet? Whats the token rate?
  
  **Tags:** gpu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-30. Score: 3 | Accepted answer
  
  memory pooling is not really much of a thing these days: the interface is not really that all of a sudden you get a single address space. You still have individual GPUs, you just specify the enablement of peer transfers. This makes sense from a design point of view, because in order to write efficient software the software has to really be aware of which data is on which physical device so that efforts can be made to optimize and reduce (unnecessary) transfers.
  
  Therefore I'd say the premise of your question is flawed. Perhaps it should be edited.
  
  Anyway, I'm running llama.cpp with dual 3090 with NVLink enabled. llama.cpp does have implemented peer transfers and they can significantly speed up inference. For example 10 tok/s -> 17 tok/s for a 70B model.
  
  > **Comments:**
  > **username_2**: I get 17.1 tok/s for a 70B model on dual 3090 WITHOUT NVLink, using a llamafile. Don't have NVLink to test  
  > **username_1**: Yes my answer is outdated, I have known for a few months now that it is indeed possible to have very little data volume shuttling back and forth to inference LLMs across two GPUs. now the GPUs aren't going to be each fully loaded, but you do leverage both their memories so you are still winning.   NVLink is useful for various training use cases. And simulations and other HPC type workloads.  
  > **username_2**: Can you report the speed you get with and without NVLink now, on something like [70B-LLama](https://huggingface.co/Mozilla/Meta-Llama-3-70B-Instruct-llamafile)?  
  > **username_1**: There isn't a way to [disable nvlink](https://forums.developer.nvidia.com/t/need-example-to-disable-nvlink/70818/4) in software if it is physically connected. I could physically disconnect it but my system setup is a bit unconventional and I can't quite do that right now, sorry. But by monitoring the data transferred via some command like `watch -n.5 nvidia-smi nvlink -gt d` you can get a good sense of how much the nvlink is being utilized; when below a rate your PCIe bus can provide (which could vary by a factor of up to 64 between x1 gen 2 and x16 gen 4!) it won't provide much benefit.  
  
  ---
  
  