To add a bit more details to @ptrblck’s answer from an NLP point of view: As you’ve rightly pointed out, there’s no clear answer, and the decision is generally task dependent. It essentially boils down to the semantic of word embeddings, and if they are meaningful in the context of your task.
The most straightforward example is sentiment analysis. In Word2Vec and Glove, to word vectors are similar if the respective words are often used in similar contexts. That means, the words like ugly and beautiful may have similar word vectors since both words are used to describe appearances. These word embeddings generally do not capture the “polarity” of words. That’s not just a guess, but I’ve noticed this first hand when I trained a Word2Vec model 16 myself, and checked the most similar words of adjectives.