DETERMINATION OF DEVICE POSE USING TEXT MATCHING FROM CAPTURED IMAGES
Both server based and client based systems can be used for precisely detecting orientation of a user device. Existing sets of geographically referenced imagery include identifiable text, and the same text may appear within multiple images within the set where the multiple images are taken from different viewpoints and camera angles. Multiple observations of the same physical text in the world are used to triangulate measurements and create a 3 dimensional (3D) location and direction for a specific text. The 3D location for the specific text is used to create a spatially indexed 3D text database, which can be located at a server and/or downloaded onto the client device. As a client device captures an image, character recognition is performed on the image, and the recognized text is compared to the 3D text database to find database text that matches the recognized text in the image. Once one or more text pieces have been matched, a triangulation computation can derive the device's orientation and location. BACKGROUND: A user device, such as a smartphone, in a dense urban environment has only a rough estimate of the location and orientation of device. This is typically derived from the sensors on the phone and GPS signals. For high precision augmented reality applications, such as overlaying geolocated content into the camera view, the orientation of the device needs to be precise to <1deg and its location to 1m. 2 Reinhardt et al. DETERMINATION OF DEVICE POSE USING TEXT MATCHING FROM CAPTURED IM Published by Technical Disclosure Commons, 2017 DETAILS: A precise 6 degree of freedom pose of a client device can be determined by comparing text in images captured by the client device with a 3 dimensional (3D) geolocated text database. The 6 degree of freedom pose may include 3 location parameters, such as x, y, z, coordinates, and 3 orientation parameters, such as yaw, pitch, and roll. As the client device captures an image, character recognition is performed on the image, and the recognized text is compared to the 3D text database to find database text that matches the recognized text in the image. Once one or more text pieces have been matched, a triangulation computation can derive the device's orientation and location. The 3D text database may be created from a large data set of geo referenced imagery. From this set, text can be identified using optical character recognition (OCR) algorithms. As there are multiple observations of the same physical text in the world it is possible to triangulate the measurements and create a 3D location and direction for a specific text. Accordingly, each specific text may be stored in association with a 3D location and direction. In its simplest form, the 3D text database is very small, with each record storing the text, xyz text pose, and an error ellipsoid. This allows the client device to load the 3D text database into memory and do the localization on the device without requiring a server connection. In other examples, the 3D text database may also store a direction normal to a plane of the text, an extent of the text (e.g. a bounding box) and other attributes, such as foreground, background, color, font, case (e.g. upper/lower case) etc. The 3D text database may be preexisting or created upfront using known information and updated from time to time. In other examples, the database may be 3 Defensive Publications Series, Art. 741 [2017] http:/www.tdcommons.org/dpubs_series/741 created over time through repeated observations from coarsely localized devices which start adding 3D text to the database, starting with very large error ellipsoids, and refining the text location over time. The client device can be any type of mobile electronic device capable of capturing and processing images. For example, the client device may be a smart phone, tablet, video game system, or the like. In some examples, the 3D text database is stored in a memory of the client device. For example, the client device may communicate with a server over a wired or wireless network, and download the 3D text database from the server. In other examples, the 3D text database may be loaded onto the client device through a storage medium such as a flash drive, USB memory device, etc. Regardless of how the 3D text database is loaded onto the client device, it may also be periodically updated. The client device captures an image of its surroundings. For example, the client device may be turned to camera mode, video mode, to an application, or to any other mode in which images are received through a camera or other image capture device of the client device. Information, specifically text, from the images is detected, such as through optical character recognition. Detected text in the image is compared to the 3D text database. For privacy reasons, the text could in some examples be encrypted with a one way algorithm, such as a hash. When matches are found for the detected text, location information from the 3D text database is associated with the detected text. Triangulation techniques are then used to determine a pose of the client device. The pose includes orientation in x, y, and z directions, as well as location in x, y, and z directions. 4 Reinhardt et al. DETERMINATION OF DEVICE POSE USING TEXT MATCHING FROM CAPTURED IM Published by Technical Disclosure Commons, 2017 Fig. 1 below illustrates an example image captured by an image capture device of a client device. The image includes three pieces of text. Those three pieces of text are encircled in Fig. 1 in order to highlight them, but will not necessarily be encircled in the captured image or in later processing of the captured image. The three pieces of text may be recognized using, for example, optical character recognition. Once recognized, they are compared to the 3D text database. Matching text in the 3D text database may have associated location information. Fig. 1 Example image captured by client device, with detected text highlighted. The location information associated with the text matches in the 3D text database includes direction information as well as positional information. For example, an x, y, z coordinate may be associated with each word, with each group of words, or with each letter. Additionally, a direction in which each word or group of words reads from left to 5 Defensive Publications Series, Art. 741 [2017] http:/www.tdcommons.org/dpubs_series/741 right is also associated with each word or group of words. For example, as shown in Fig. 2 below, a first piece of text "Town Square" extends in a first direction, a second piece of text "Barbeque Restaurant" extends in a second direction, and a third piece of text "323" extends in a third direction. In this example, each of the first, second, and third directions are different, though in other examples directional information for two or more pieces of text may be the same. The directional information may also be defined along an x,y,z axis, using vectors, or by any other mechanism for indicating direction. According to some examples, directional information of the detected text may also be determined from the captured image. For example, vector analysis may be used to determine a direction for the detected text, and use that information to determine the client device orientation. As another example, a size of the detected text may appear to become smaller as the text becomes further away, thereby indicating direction. As yet another example, an orientation of the text can help to determine whether it is being read from the front or the back. 6 Reinhardt et al. DETERMINATION OF DEVICE POSE USING TEXT MATCHING FROM CAPTURED IM Published by Technical Disclosure Commons, 2017 Fig. 2: Example of directional text information used to determine client device location and orientation. Using the location information, including position and directional information as shown in the example of Fig. 2, a pose of the client device is determined. Such pose may include both location and orientation (yaw, pitch, roll) The pose may be determined using triangulation techniques. Information regarding the image capture device, if known, may assist in the determination of pose information from the detected text. For example, if a focal length of the image capture device is known, assumptions may be made regarding the detected text. As an example of an assumption, information such as a distance between the image capture device and the detected text may be assumed. Using this information, less points of detected text are needed to determine a pose of the image capture device. By way of example only, whereas five pieces of text may be needed to determine the pose without knowing any information regarding the image capture device, only three or four pieces of text may be needed if the focal length of the image capture device is known. 7 Defensive Publications Series, Art. 741 [2017] http:/www.tdcommons.org/dpubs_series/741 In some examples, the client device may be trained to recognize logos, and use the logos to also determine pose. For example, the image capture device of the client device may detect a symbol representing a particular store, brand, etc. The symbol may be included in the 3D text database along with an associated location. Accordingly, the logo may be used in addition or in the alternative to text. In instances where only one or two pieces of text are visible, but several logos are more clearly visible, this may help to compute a more accurate pose. The pose information may be determined to such a degree of accuracy and precision that such information can be used in other applications, such as augmented reality applications. In such applications, an optimal user experience is provided with a high degree of precision. For example, if the application is placing a virtual arrow pointing to a doorway, or a virtual label on a building, such virtual features will only be correctly placed if the cli