Improve installation + code snippets

by Xenova HF staff - opened Jul 23

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+29

-57

Xenova

Hugging Quants org Jul 23

No description provided.

Improve installation + code snippetse4beb03d

pcuenq

Hugging Quants org Jul 23

Looks good to me! Should the dtype be bfloat16, or not when using AWQ?

alvarobartt

Hugging Quants org Jul 23

LGTM, thanks @Xenova ! And @pcuenq apparently for AWQ using float16 is recommended, if you set the torch_dtype=torch.bfloat16 the following message appears We suggest you to set torch_dtype=torch.float16 for better efficiency with AWQ., this is why we're using torch.float16 with AWQ :D

alvarobartt changed pull request status to merged Jul 23

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment