content.raw: I'm (finally) releasing a Python script that trims excess weights in Gemma2 full-weight models that bloated by ~1B parameters due to an early mergekit bug.
content.type: text
content.value: I'm (finally) releasing a Python script that trims excess weights in Gemma2 full-weight models that bloated by ~1B parameters due to an early mergekit bug.
content.type: new_line
content.href: https://github.com/jim-plus/Gemma2-mergekit-remediation
content.raw: https://github.com/jim-plus/Gemma2-mergekit-remediation
content.type: link
content.type: new_line
content.type: new_line
content.raw: I'd noticed something was off when merges of Gemma2 9B models ended up having ~10B parameters. The current mergekit package is fine, but there are still bloated models on HF that could stand to be fixed.
content.type: text
content.value: I'd noticed something was off when merges of Gemma2 9B models ended up having ~10B parameters. The current mergekit package is fine, but there are still bloated models on HF that could stand to be fixed.
content.type: new_line
content.type: new_line
content.raw: The script assumes that it will be run from the same directory as the model weights, and will trim the unnecessary lm_head.weight tensor and corresponding index entry.
content.type: text
content.value: The script assumes that it will be run from the same directory as the model weights, and will trim the unnecessary lm_head.weight tensor and corresponding index entry.
rawContent: I'm (finally) releasing a Python script that trims excess weights in Gemma2 full-weight models that bloated by ~1B parameters due to an early mergekit bug.
  https://github.com/jim-plus/Gemma2-mergekit-remediation
  
  I'd noticed something was off when merges of Gemma2 9B models ended up having ~10B parameters. The current mergekit package is fine, but there are still bloated models on HF that could stand to be fixed.
  
  The script assumes that it will be run from the same directory as the model weights, and will trim the unnecessary lm_head.weight tensor and corresponding index entry.
reactions.reaction: 🔥
reactions.users: John6666 Bredvige clem iky1e
reactions.reaction: 👍
reactions.users: timmylai agentlans WaveCut
reactions.reaction: ❤️
reactions.users: asach
commentators._id: 663a18c876e6d5b98f3cc94d
commentators.avatarUrl: /avatars/faa2a47dbdd3bce6d47e0fbed6170627.svg
commentators.fullname: Alan Tseng
commentators.name: agentlans
commentators.type: user
commentators._id: 65c992424936ab38ecf706b0
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c992424936ab38ecf706b0/aq7vuHFPO1S93fwJk0Cuq.jpeg
commentators.fullname: Jim Lai
commentators.name: grimjim
commentators.type: user