getting a couple of errors " No model.safetensors.index.json found in remote" and "assert hidden_states.dtype == torch.bfloat16, "hidden_states must be bfloat16"
getting a couple of errors " No model.safetensors.index.json found in remote" and "assert hidden_states.dtype == torch.bfloat16, "hidden_states must be bfloat16"
Symptoms:
Assertion failures and hidden-state mismatches when loading models with recent versions of the transformers library.
Root Cause: A guard statement introduced upstream now enforces stricter type alignment for tensor dtypes (bfloat16 → fp16).
Resolution:
Two mutually-exclusive upgrade paths are available:
Full Re-download (Recommended)
Delete the local model directory.
Re-pull the latest weights from the Hugging Face repository (main branch).
No further action required.In-place Patch (Lightweight fallback)
Run python fp16.py located in the repository root.
The script performs an in-place dtype cast from bfloat16 to fp16 using sequential on-loading, eliminating the need for additional GPU memory or large RAM buffers.
Verify SHA-256 checksums after completion to ensure integrity.
Both paths resolve the compatibility issue; choose whichever option best aligns with your bandwidth and compute constraints.