Hello @afractalthought,
You can try Sentence transformer which is much better for clustering from feature extraction than vanilla BERT or RoBERTa. When applying cosine similarity on the sentence  embedding from this model, documents with semantic similarity should get a higher similarity score and clustering should get better.