Does dimension reduction in more than 2 (or 3) dimension make sense? I'm using dimension reduction for data analysis (pca, tsne, umap...). Most examples I see project data in only 2 (or 3) dimensions, but I would naively imagine that by projecting in more dimension and visualize those dimensions 2 by 2 on multiple plots I could see more sub-characteristics of the data.
For example, if I have a dataset of cats and dogs data, and that after projecting it in 2dim I see mainly two cluster corresponding two the two species: is there a chance that if I project my data in 4 dimension and plot 2 graph (one with embeddings dimensions 1 and 2 and the other with dimensions 3 and 4) I would be able to see 2 clusters corresponding to cat and dogs on first graph and clusters corresponding to dog sub-species (like Labrador and bulldogs) on the second graph ?
So in other words, is it worth projecting my data in more than 3 dimensions with those kind of algorithms ?

Projecting to a low number of dimensions is mostly to help visualize the information for humans (as you suggest, there's absolutely no reason higher dimensions might not be better otherwise for many cases).
There are a lot of examples of representing things in higher embedding spaces, e.g.

*

*the word2vec embeddings for words (see here or here - similar ideas are in a sense used in modern transformer neural network language models) that often use about 100 to 300 dimensions,

*embeddings for categorical data (like capturing the properties of each shop of a retailer for which in the example 10 dimensions were used and then projected to 2D for plotting; see also the section on category embeddings in a popular recent Deep Learning book, which I think also offers a rule of thumb for how to choose the dimension),

*or each row of a dataset (e.g. through a denoising autoencoder in the example I think a few 1000 dimensions were used).

