vllm-sr

/

d3-flash

Like 101

Follow

vLLM Semantic Router 600

Hi @FaisaI ! Could you tell us what you need the extra GPUs for? The answer depends on it.

Is it more throughput, with a full copy of the model on each GPU (DP), or does d3-flash not fit on one of your GPUs, so it has to be split across them (TP)?

Which GPUs are you using?

Hi Subin!

Thanks for your reply!

Yes, mainly the tensor parallelism, since unfortunately, D3-Flash doesn't fit on my A4000 GPU unless I use the quantized version of it.

Thanks!

Thanks! We don't have a TP number for d3-flash yet. On a single 16 GB card it does run in BF16 with vLLM's prefetch offload, which keeps a quarter of the layers in CPU memory and streams them in:

vllm serve vllm-sr/d3-flash --offload-group-size 4 --offload-num-in-group 1 --gpu-memory-utilization 0.9 --enforce-eager --max-model-len 4096 --limit-mm-per-prompt '{"image":0,"video":0}'

On an RTX 4070 Ti SUPER (16 GB) that gave about 350 ms per request and about 13 requests/s with 16 concurrent clients, and its answers matched the model's own system_one(). That was text only, so images may need more room. TP across two A4000s should also work, but we haven't measured it.

Will do thanks!