Doing this works fine (which is exactly the same as done in the Inference API). @Narsil this seems like an issue in the deployed image (?). Is there any chance an old version was deployed by accident?
!pip install -e git+https://github.com/UKPLab/sentence-transformers@v2_dev#egg=sentence-transformers
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("flax-sentence-embeddings/all_datasets_v3_distilroberta-base")