Effects of class imbalance on neural network weights My question is about unbalanced classes problem in case of a classifier neural network for natural language processing (in particular, a neural network with LSTM).
I want to train a neural network to discriminate data between four classes and then I want to extract weights vectors from the embedding layer so that I can use them to have a numerical dense vector representation of my data.
If the classes are unbalanced (for example, class 1 has 27000 data, class 2 has 3600 data, class 3 has 260 data and class 4 has 10 data) and the metrics is set to "accuracy" (imaging to develop it using Keras in Python), the problem is that our classifier will tend to classify wrong the classes with less data so to reach a high accuracy. However, I do not really focus one thing: how are weights influenced by this fact ? For example, if I extract weights vectors that represent data from my embedding layer and I plot them, because of imbalanced classes problem, the effect is that data are not well discriminated in the plot according to the different classes to which they belong ? I mean: in this plot I should observe four clusters but they are all overlapping because of the unbalanced classes problem ?
Maybe it is a trivial question, so I apologize and I thank you in advance.

First to your plot: the embeddings may be being further transformed farther down the network. Further layers in the network may be transforming the values of your embedding layer in a way that is not emendable to the plotting algorithm you are using. t-SNE plots usually work pretty good, but aren't perfect. In other words, just because the plot doesn't discriminate doesn't mean the model is failing.
However, you did say the classification isn't working well because of the imbalance. I want to make a slight distinction here: you actually have two problems: (a) imbalance, and (b) lack of data. Because there is a lack of data, your network doesn't have enough information to create an abstraction of the features of the class. There isn't really much to do about that; you just need more data.
You can address the imbalance problem though. To do this, use cross-entropy as a loss function instead of accuracy. Cross entropy fixes the problem of imbalanced classes. You can learn more about it here. However, with your data set, you simply don't have enough data for classes 3 and 4 to reasonably train a LSTM network.
