Removing instances that decrease accuracy of Machine learning algorithm Methodology Is it bad practice to run a Machine learning algorithm on an experimental dataset, check the MAE, and remove the instances that have a value of MAE above a certain limit? If we run the algorithm without those instances the accuracy increases significantly of course.
I know that something similar can be used when we want the ML to learn from its mistakes. Would it be better to make my own loss function? At the moment I am using a regression NN from keras and tensorflow.
I have reached the point where no matter the model, my accuracy doesn't increase anymore. I know that there might be outliers in my dataset but detecting them is not straightforward at all.
I am relatively new in the Machine Learning field so I will be very grateful for any advice, suggestions, or corrections.

If you're systematically removing the hardest examples from your data set (the ones that your model gets wrong, i.e. the largest values), you're not getting a good estimate of how the model will perform in reality. You're correct that removing the largest numbers from a list will have a smaller mean, but that smaller mean doesn't have any bearing on how your model does.
Some problems are simply challenging, perhaps because there's a large degree of randomness in the process, or because the features don't strongly bear on what you want to predict, or another reason.
