Sounds good!
I found two solvenian model on the hub, which you can give a try to fine-tune and compare with your bag-of-words method:

EMBEDDIA/sloberta · Hugging Face 3
EMBEDDIA/crosloengual-bert · Hugging Face 2