content.raw: Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
content.type: text
content.value: Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
content.type: new_line
content.type: new_line
content.raw: The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
content.type: text
content.value: The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
content.type: new_line
content.type: new_line
content.raw: Had claude make me a script, using the new Reflection-70B, and these are the results:
content.type: text
content.value: Had claude make me a script, using the new Reflection-70B, and these are the results:
content.type: new_line
content.type: new_line
content.raw: Total weights: 70553706496
content.type: text
content.value: Total weights: 70553706496
content.type: new_line
content.raw: Fully representable: 70530215524
content.type: text
content.value: Fully representable: 70530215524
content.type: new_line
content.raw: Squashed: 23490972
content.type: text
content.value: Squashed: 23490972
content.type: new_line
content.raw: Percentage squashed: 0.03%
content.type: text
content.value: Percentage squashed: 0.03%
content.type: new_line
content.type: new_line
content.raw: 0.03%!!!!
content.type: text
content.value: 0.03%!!!!
content.type: new_line
content.type: new_line
content.raw: A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
content.type: text
content.value: A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
content.type: new_line
content.type: new_line
content.raw: This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
content.type: text
content.value: This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
content.type: new_line
content.type: new_line
content.raw: Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
content.type: text
content.value: Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
rawContent: Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
  
  The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
  
  Had claude make me a script, using the new Reflection-70B, and these are the results:
  
  Total weights: 70553706496
  Fully representable: 70530215524
  Squashed: 23490972
  Percentage squashed: 0.03%
  
  0.03%!!!!
  
  A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
  
  This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
  
  Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
reactions.reaction: 🔥
reactions.users: prithivMLmods Sri-Vigneshwar-DJ Joseph717171 not-lain John6666 morph3v5
reactions.reaction: ❤️
reactions.users: Joseph717171 KhaldiAbderrhmane nicoboss
reactions.reaction: 👍
reactions.users: MoonRide blotfaba nicoboss
reactions.reaction: 👀
reactions.users: John6666 Joseph717171
reactions.reaction: 🤯
reactions.users: louisbrulenaudet Joseph717171
reactions.reaction: 🤗
reactions.users: Joseph717171
reactions.reaction: 🧠
reactions.users: Joseph717171
reactions.reaction: 🚀
reactions.users: Joseph717171
commentators._id: 64a9896531a43d726502b9f5
commentators.avatarUrl: /avatars/da52e5fce67042332fa1e9f5fd3e5635.svg
commentators.fullname: Luke Chadwick
commentators.name: vertis
commentators.type: user
commentators._id: 6435718aaaef013d1aec3b8b
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6435718aaaef013d1aec3b8b/XKf-8MA47tjVAM6SCX0MP.jpeg
commentators.fullname: Bartowski
commentators.name: bartowski
commentators.type: user
commentators._id: 6640bbd0220cfa8cbfdce080
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6640bbd0220cfa8cbfdce080/wiAHUu5ewawyipNs0YFBR.png
commentators.fullname: John Smith
commentators.name: John6666
commentators.type: user
commentators._id: 63c1dfd4a0ffa3857eb362a9
commentators.avatarUrl: /avatars/ea4398745974d781ae9dc0e95b12cabe.svg
commentators.fullname: Joseph
commentators.name: Joseph717171
commentators.type: user
commentators._id: 666b14d48ccf7f0aa469f3bd
commentators.avatarUrl: /avatars/99351620d65d263418e6d0d4e170f055.svg
commentators.fullname: Abrosimov
commentators.name: ajiriro
commentators.type: user
commentators._id: 66d1efa935c36f266f507cff
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66d1efa935c36f266f507cff/a2-fPLeGwAp5fqCKdqfzp.jpeg
commentators.fullname: Harmendo
commentators.name: Hampetiudo
commentators.type: user
commentators._id: 63ef9ccae2319f35318ca8de
commentators.avatarUrl: /avatars/3b03217c22442b7bfed9beac2bf50d17.svg
commentators.fullname: Alex Daminger
commentators.name: Handgun1773
commentators.type: user
commentators._id: 6570493c84632c95659d342a
commentators.avatarUrl: /avatars/98d7cbc7bf4cbf4f2810cbc0a1a34d64.svg
commentators.fullname: Iwan Kawrakow
commentators.name: ikawrakow
commentators.type: user
commentators._id: 655e9450d5c0d3db536859d1
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4Az8a8F60rNOD3L3ThsCe.png
commentators.fullname: Compilade
commentators.name: compilade
commentators.type: user