repulled weights but getting this error assert hidden_states.dtype == torch.bfloat16, "hidden_states must be bfloat16"
#5
by
bluenevus
- opened
A little context. only reason I was looking for fp16 was to run this on some old rtx8000 which can't do bf16.
I'm gettin this error assert hidden_states.dtype == torch.bfloat16, "hidden_states must be bfloat16"
using sglang
Python script in repo will fix the issue just edit it to point at your model files directory.
I’ll have new model files up in the next 48 hours.