The present invention provides methods and systems to extract primary and secondary motions in augmented reality systems. The method for motion estimation in an augmented reality (AR) system includesthe steps of: receiving inertial sensor data and image data during movement of the AR system generating a probability map based on the inertial sensor data and the image data, the probability map corresponding to one frame in the image data and including probability values indicating that each pixel in the one frame is in an inertial coordinate frame or a local coordinate frame with a convolutional neural network encoder/decoder, identifying visual observations of at least one landmark in the local coordinate frame based on the image data and the probability map, and generating an estimate ofsecondary motion in the local coordinate frame based on a first prior state in a hidden Markov model (HMM) corresponding to the local coordinate frame and the visual observations of the at least one landmark in the local coordinate frame.
Methods and systems to extract primary and secondary motions in augmented reality systems