content.raw: So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
content.type: text
content.value: So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
content.type: new_line
content.type: new_line
content.raw: It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
content.type: text
content.value: It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
content.type: new_line
content.type: new_line
content.raw: However what the quantization then does with that information is where I was wrong.
content.type: text
content.value: However what the quantization then does with that information is where I was wrong.
content.type: new_line
content.type: new_line
content.raw: I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
content.type: text
content.value: I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
content.type: new_line
content.type: new_line
content.raw: Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
content.type: text
content.value: Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
content.type: new_line
content.type: new_line
content.raw: The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
content.type: text
content.value: The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
content.type: new_line
content.type: new_line
content.raw: But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
content.type: text
content.value: But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
content.type: new_line
content.type: new_line
content.raw: Huge shoutout to 
content.type: text
content.value: Huge shoutout to 
content.raw: @compilade
content.type: mention
content.user: compilade
content.raw:  for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
content.type: text
content.value:  for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
rawContent: So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
  
  It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
  
  However what the quantization then does with that information is where I was wrong.
  
  I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
  
  Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
  
  The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
  
  But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
  
  Huge shoutout to @compilade for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
reactions.reaction: ❤️
reactions.users: MaziyarPanahi victor Themoneybuddha John6666 bullerwins Rybens Anderson452 NOBODYVONNOTHING qnixsynapse Clausss exdysa AtAndDev IAmTheCollector wsuff rreed-pha djuna MarinaraSpaghetti Pomni RakshitAralimatti osanseviero venketh Arakinas wonderboy nicoboss Joseph717171 AIGUYCONTENT Handgun1773 bradeenl EloyOn
reactions.reaction: 👍
reactions.users: Aurelien-Morgan Clausss AtAndDev sroecker iandol rreed-pha ajibawa-2023 ijohn07 thiera1 nbeerbower newsletter venketh mpk63 wonderboy clem nicoboss jsfs11 Joseph717171 waldie fendiprime
reactions.reaction: 🚀
reactions.users: rreed-pha wonderboy nicoboss Joseph717171
commentators._id: 5fd5e18a90b6dc4633f6d292
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd5e18a90b6dc4633f6d292/gZXHW5dd9R86AV9LMZ--y.png
commentators.fullname: Maziyar Panahi
commentators.name: MaziyarPanahi
commentators.type: user
commentators._id: 66bfd0deef1f45cd717f88fa
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_O-ycBVEVTAvhqC9wH4v2.png
commentators.fullname: Eric Sims
commentators.name: Themoneybuddha
commentators.type: user
commentators._id: 631680bf3edbcfbc5d490227
commentators.avatarUrl: /avatars/c6c710651ec2bde1680497afd4649281.svg
commentators.fullname: John Freier
commentators.name: mpk63
commentators.type: user
commentators._id: 623aada982038110d90921f3
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623aada982038110d90921f3/nSHSXJz7PYyw9fn8qoAn4.jpeg
commentators.fullname: zero
commentators.name: wonderboy
commentators.type: user
commentators._id: 63c1dfd4a0ffa3857eb362a9
commentators.avatarUrl: /avatars/ea4398745974d781ae9dc0e95b12cabe.svg
commentators.fullname: Joseph
commentators.name: Joseph717171
commentators.type: user