A near-eye display (NED) device, such as a head mounted display (HMD) device, may be worn by a user for an augmented reality (AR) experience, mixed reality or a virtual reality (VR) experience. For playing games or other activities in which a user or virtual objects move around a user environment, a three-dimensional (3D), detailed mapping of the environment identifying the shapes and positions of real objects in a location can be used to identify where virtual objects can be placed and moved as if they were real. For example, a real object does not run through the couch, and a user seeing a virtual object do so receives a distracting unrealistic depth cue interrupting the AR or VR experience. A detailed mapping like a very dense 3D surface mesh representation of a space allows for realistic experiences but is computationally expensive, particularly for computer resources available in a near-eye display (NED) system.