Unbalanced data: Like, you have more  image examples with their corresponding labels at certain (x, y) locations than some other (x, y) locations. This will, probably, make your classifier more biased to the examples that appear more than others.
Loss function: CrossEntropy is highly recommended.