Howto run nvidia/NV-Embed-v2 on GPU
#26
by
hozbey
- opened
Hello,
I am using "nvidia/NV-Embed-v2" with Open-WebUI for RAG(Retrieval-augmented generation). But the Model is running on CPU.
Would you please comment that how could I run "nvidia/NV-Embed-v2" on GPU resources?
Best Regards
Huseyin A. Ozbey
load like this
model_embed_nvidia = AutoModel.from_pretrained(
f"{MOUNT_DIR}/NV-Embed-v2", device_map={"": "cuda:0"}, trust_remote_code=True
)