Hi @cezary, since you want to cluster articles you could use any of the “encoder” Transformers (e.g. BERT-base) to extract the hidden states per article (see e.g. here 194 for an example with IMDB) and then apply clustering / dimensionality reduction on the hidden states to identify the clusters.
If you’re dealing with an unsupervised task, I’ve found that UMAP 136 + HDBSCAN 75 works well for the second step.
HTH!