Improving model accuracy by manually labelling more data? I'm working on a binary classification problem with a small dataset n < 2000 that predominantly uses text data. Let's say the model tends to misclassify observations where a certain categorical column value is 'A' or 'B' but performs well on 'C' or 'D'.
If I have the means to get help with manual data labelling, is it important that I specifically get more observations that have column value 'A' or 'B', or does that not matter/add bias to the system?

You could also try active learning methods which will use your current classifier to automatically estimate which additional data would be most informative for it if labeled next. There are practical open-source implementations of this, eg. some methods for Python I helped develop.
You are currently using your own intuition to assume collecting more data from 'A' or 'B' will be best, but perhaps this is still sub-optimal. Active learning can help you figure this out
