A colleague of mine has figured out a way to work around this issue. Although both Huggingface and Fairseq use spm from google, the tokenizer in Fairseq map the id from spm to the token id in the dict.txt file, while Huggingface’s does not
We will have to write a custom Tokenizer in Huggingface to simulate the behavior as in Fairseq