What is your main goal? Is it to have a classifier for  toxic vs non-toxic sentences? Or to cluster sentence embeddings by semantic meaning?
For creating sentence embeddings I would recommend sentence transformers 9. It’s an extensions of regular huggingface transformers but optimized for creating text embeddings.