After more investigating, I managed to make the rebase work, using this script:
huggingface-cli login
huggingface-cli repo create ${MODEL_NAME} --organization ${NEW_ORG}
git lfs install --skip-smudge
git clone https://huggingface.co/${NEW_ORG}/${MODEL_NAME}
cd ${MODEL_NAME}
git remote add upstream https://huggingface.co/${ORIGINAL_ORG}/${MODEL_NAME}
git fetch upstream
git rebase upstream/main
git push --force-with-lease

I thought I had solved the case… Except that the new model is somehow forbidden when I tried to load it in my code:
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://cdn-lfs.huggingface.co/DataikuNLP/average_word_embeddings_glove.6B.300d/d819348e583fca49cf3980e34505d52a3f842064ebd9dc255484125357771240

Is this issue solvable from my end or does it require a change on your model hub infrastructure?
Thanks for your help,
Alex