I have been using sentence-transformers to calculate document embeddings and then used them as input for document clustering.
I read somewhere that it is best to use a model that was trained on a downstream task that focuses on text similarity. But a standard Bert model worked good enough in my use case.
Moreover, you can play around with different approaches how the embeddings are extracted, like mean-pooling, [CLS] token, … I never had a metric to compare if one is better than the other - I just did some cluster analysis and looked visually if the clusters made sense.
Here is a small working example (using a German dataset with a German Bert model):
pip install -U -q sentence-transformers

from datasets import load_dataset
from sentence_transformers import SentenceTransformer
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

dataset = load_dataset("gnad10", split='train[:10%]')
print(dataset.num_rows) # 924

model = SentenceTransformer('deepset/gbert-base')
embeddings = model.encode(dataset["text"])
print(embeddings.shape) # (924, 768)

result = TSNE().fit_transform(embeddings)
print(result.shape)  # (924, 2)

plt.scatter(result[:, 0], result[:, 1])