Effects of class imbalance on neural network weights My question is about unbalanced classes problem in case of a classifier neural network for natural language processing (in particular, a neural network with LSTM).
I want to train a neural network to discriminate data between four classes and then I want to extract weights vectors from the embedding layer so that I can use them to have a numerical dense vector representation of my data.
If the classes are unbalanced (for example, class 1 has 27000 data, class 2 has 3600 data, class 3 has 260 data and class 4 has 10 data) and the metrics is set to "accuracy" (imaging to develop it using Keras in Python), the problem is that our classifier will tend to classify wrong the classes with less data so to reach a high accuracy. However, I do not really focus one thing: how are weights influenced by this fact ? For example, if I extract weights vectors that represent data from my embedding layer and I plot them, because of imbalanced classes problem, the effect is that data are not well discriminated in the plot according to the different classes to which they belong ? I mean: in this plot I should observe four clusters but they are all overlapping because of the unbalanced classes problem ?
Maybe it is a trivial question, so I apologize and I thank you in advance.

You won't fully overcome your class imbalance by using an appropriate loss function alone. The network will produce a better result by almost always ignoring class 4 unless it's very easy to classify and the network can be very confident, however cross entropy will ensure that class 4 isn't predicted completely wrong because a 0.0 prediction for a 1.0 label is infinite error, or "surprise", but that doesn't preclude it from always giving class 4 a more reasonable value of say 0.3 when it's uncertain, the imbalance is likely to produce this kind of result with cross entropy. You could use that to adjust the classification, but I think a simpler approach is oversampling.
Oversampling your under represented class during training is a common approach. For example if your batch size is 8, two of those values come from each class. As was mentioned by Tanner, you have very little data in class 4, and class 3 for that matter, so now you have an issue of over fitting for class 4 (I would expect class 4 to generalize less well than class 1 and class 2). To improve that situation you should take any steps you can to augment your class 4 &
3 data. Data augmentation means you add random permutations to your data. What you add to the data should reflect actual variation you are likely to see in your real world data. Simply adding Gaussian noise to your class 4 features isn't likely to produce substantially better results. Good data augmentation requires you understand your data.
