Hi @ptrblck,
This is the link to the Fairseq GitHub issue 2.
As a next step, I will be doing benchmarking of my code using the preprocessing module vs raw text on small dataset such that I can see if there will be any impact on the accuracy or not. I will provide the update soon here. This may help the community.
Thanks