I added support for KenLM using the flashlight library here: Wav2Vec-Wrapper/test.py at main · Edresson/Wav2Vec-Wrapper · GitHub 203
It supports the use of the binary file instead of the arpa and it is also possible to restrict the model’s vocabulary.