Moritz Laurer

Hey @borowis , I don't think there is a plan to add embedding models to the NIM API. Embedding models are quite small which makes them easier to run on accessible hardware (vs. the H100 GPUs running the large LLMs on the NIM API). I'd recommend using a cheap GPU (or even a CPU) via the HF dedicated endpoints for deploying embedding models: https://huggingface.co/inference-endpoints/dedicated And you can use the autoscaling/scale-to-zero feature to avoid unnecessary costs
(The smaller BGE models from the MTEB leaderboard are always a good place to start)

The reference names are always in this collection for the NIM API: https://huggingface.co/collections/nvidia/nim-serverless-inference-api-66a3c6fcdcb5bbc6e975b508

It works for me with this code actually. The small change I made is a minimal change in the model identifier (removing the "meta"). The issue is that Meta renamed their models on HF recently.
It works for me with this code. Can you try again with this code?

#!pip install "huggingface_hub>=0.25.0"
from huggingface_hub import InferenceClient


client = InferenceClient(
    base_url="https://huggingface.co/api/integrations/dgx/v1",
    api_key=os.getenv("HF_TOKEN_ENTERPRISE")  # see docs: https://huggingface.co/blog/inference-dgx-cloud#create-a-fine-grained-token
)

output = client.chat.completions.create(
    model="meta-llama/Llama-3.1-405B-Instruct-FP8",  #"meta-llama/Meta-Llama-3.1-405B-Instruct-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Count to 10"},
    ],
    max_tokens=1024,
)

print(output)

ok, I could reproduce the issue and I get the same error, I'm reporting it to Nvidia, thanks for reporting

What error do you get for the 405b?

Hey @jmparejaz , did you use a token from an enterprise org as explained here https://huggingface.co/blog/inference-dgx-cloud#create-a-fine-grained-token ?

Here is the full thesis if you're interested: https://research.vu.nl/ws/portalfiles/portal/355675396/dissertationlaurerfinal+-+66c885c7e9d0b.pdf
 
Here is the collection of my most recent models: https://huggingface.co/collections/MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f

#!pip install "huggingface_hub>=0.25.0"
from huggingface_hub import InferenceClient

client = InferenceClient(
    base_url="https://huggingface.co/api/integrations/dgx/v1",
    api_key="MY_FINEGRAINED_ENTERPRISE_ORG_TOKEN"  # see docs: https://huggingface.co/blog/inference-dgx-cloud#create-a-fine-grained-token
)

output = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-405B-Instruct-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Count to 10"},
    ],
    max_tokens=1024,
)

print(output)