content.raw: Looks like Q4_0_N_M file types are going away
content.type: text
content.value: Looks like Q4_0_N_M file types are going away
content.type: new_line
content.type: new_line
content.raw: Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
content.type: text
content.value: Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
content.type: new_line
content.type: new_line
content.raw: You can see the reference PR here:
content.type: text
content.value: You can see the reference PR here:
content.type: new_line
content.type: new_line
content.href: https://github.com/ggerganov/llama.cpp/pull/10446
content.raw: https://github.com/ggerganov/llama.cpp/pull/10446
content.type: link
content.type: new_line
content.type: new_line
content.raw: So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
content.type: text
content.value: So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
content.type: new_line
content.type: new_line
content.raw: As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
content.type: text
content.value: As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
content.type: new_line
content.type: new_line
content.raw: Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: 
content.type: text
content.value: Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: 
content.href: https://github.com/ggerganov/llama.cpp/pull/10541
content.raw: https://github.com/ggerganov/llama.cpp/pull/10541
content.type: link
content.type: new_line
content.type: new_line
content.raw: Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
content.type: text
content.value: Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
rawContent: Looks like Q4_0_N_M file types are going away
  
  Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
  
  You can see the reference PR here:
  
  https://github.com/ggerganov/llama.cpp/pull/10446
  
  So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
  
  As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
  
  Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
  
  Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
reactions.reaction: 👍
reactions.users: John6666 Felladrin agh123 RickyPossum Green4el stormchaser jupiterbjy urtuuuu Yuma42 robbiemu KeenZZZ ngxson ONTHEREDTEAM TheDrunkenSnail aviallon thelivingded bre1470
reactions.reaction: 👀
reactions.users: julien-c John6666 stormchaser ngxson EloyOn
commentators._id: 65a5ad3c0b5704678a8612b9
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a5ad3c0b5704678a8612b9/Fh6TDWyf9EGt6U2_G-v4Q.png
commentators.fullname: AItomek ;P
commentators.name: altomek
commentators.type: user
commentators._id: 6435718aaaef013d1aec3b8b
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6435718aaaef013d1aec3b8b/XKf-8MA47tjVAM6SCX0MP.jpeg
commentators.fullname: Bartowski
commentators.name: bartowski
commentators.type: user
commentators._id: 63cd4b6d1c8a5d1d7d76a778
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63cd4b6d1c8a5d1d7d76a778/FAnsBh9h5-wG9Vb252ZNt.png
commentators.fullname: Concedo
commentators.name: concedo
commentators.type: user
commentators._id: 667e18a1fd5220e7b99dffcf
commentators.avatarUrl: /avatars/07ab998c22bdc1f9824271df799d8c14.svg
commentators.fullname: Jun Young Baek
commentators.name: jupiterbjy
commentators.type: user
commentators._id: 63cf44747332dafae2b92fd8
commentators.avatarUrl: /avatars/afb6faa09e54cbde7ee0b8b6e1964236.svg
commentators.fullname: rtuuuuuuuur
commentators.name: urtuuuu
commentators.type: user
commentators._id: 65cd56e78720e05cceea9f09
commentators.avatarUrl: /avatars/8a71bab1999a8ae539fde42121abdab0.svg
commentators.fullname: Yunus Manti
commentators.name: Yuma42
commentators.type: user
commentators._id: 631396705b533c7c37a99014
commentators.avatarUrl: /avatars/2cc5f6ee43e3648726fb3cf833e34dab.svg
commentators.fullname: Venketh
commentators.name: venketh
commentators.type: user
commentators._id: 64eee9b7b3206d23d73f8dab
commentators.avatarUrl: /avatars/960c80cdc44c81721c3db0e5c2a25eed.svg
commentators.fullname: Hangou Fanou
commentators.name: Hugs4Llamas
commentators.type: user