Solved by LeviViana in post #2 
              
              
                Maybe these two questions can make things clearer: 

In your application, what is the most important metric ?
In production, what is the distribution you expect to encounter (balanced or imbalanced) ?

My guess is that you want to apply early stopping for the metric the most matters under the distri…