Sensor drifts and modelling mismatches are key factors that influence the accuracy of state of charge (SOC) estimation for LiFePO4 batteries. In this study, an observer robust to these factors is proposed. First, the causes of SOC errors, for example, modelling error and uncertain initial error, are studied. Second, a geometry classifier is designed to categorize these errors into different groups using the information of voltage error between model and measurements. Third, with the classifier, different types of errors are treated differently by switching the gains of the observer. Finally, the method is tested in comparison to the existing methods for both new and aged cells. The test results show that the proposed method can correctly categorize the error causes and take the corresponding countermeasures. The common problems encountered in SOC estimations, such as local model inaccuracy, current sensor drifting and data saturation, could be overcome. The computation time of the proposed method is close to that of the Luenberger observer, making it suitable for real embedded applications.