text:
US9405431B2 - Generating position information employing an imager - Google Patents
Generating position information employing an imager Download PDF
David L. Henty
I-INTERACTIVE LLC
2014-09-19 Application filed by I-INTERACTIVE LLC filed Critical I-INTERACTIVE LLC
2014-12-08 Assigned to I-INTERACTIVE LLC reassignment I-INTERACTIVE LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HENTY, DAVID L.
G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
G06F3/002—Specific input/output arrangements not covered by G06F3/02 - G06F3/16, e.g. facsimile, microfilm
G06F3/005—Input arrangements through a video camera
G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
G06F3/0304—Detection arrangements using opto-electronic means
G06F3/0317—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
G06F3/0386—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry for light pen
G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
G06F3/0482—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance interaction with lists of selectable items, e.g. menus
G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object or an image, setting a parameter value or selecting a range
G06F3/0485—Scrolling or panning
G06F3/04855—Interaction with scrollbars
G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
G08C23/00—Non-electrical signal transmission systems, e.g. optical systems
G08C23/04—Non-electrical signal transmission systems, e.g. optical systems using light waves, e.g. infra-red
H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
H04N21/42222—Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
H04N21/4223—Cameras
H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
H04N5/4403—User interfaces for controlling a television receiver or set top box [STB] through a remote control device, e.g. graphical user interfaces [GUI]; Remote control devices therefor
H04N5/44582—Receiver circuitry for displaying additional information the additional information being controlled by a remote control apparatus
G08C2201/00—Transmission systems of control signals via wireless link
G08C2201/30—User interface
G08C2201/32—Remote control based on movements, attitude of remote control device
G08C2201/90—Additional features
G08C2201/93—Remote control using other portable devices, e.g. mobile phone, PDA, laptop
H04N2005/4405—Hardware details of remote control devices
H04N2005/4428—Non-standard components, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone, battery charging device
H04N2005/4432—Remote control devices equipped or combined with PC-like input means, e.g. voice recognition or pointing device
H04N21/42225—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details characterized by types of remote control, e.g. universal remote control
An application control approach using generated position information of a control device and adapted for use with a display, such as a monitor or TV, is disclosed. The control device may be conveniently held by a user. An imager is employed to image the control device to detect reference fields and generate position information. This information is used by the control device to control an application.
RELATED APPLICATION INFORMATION
The present application is a Continuation of application Ser. No. 13/908,900 filed Jun. 3, 2013, U.S. Pat. No. 8,842,186, which is a Continuation-in-Part of application Ser. No. 11/255,647 filed Oct. 21, 2005, U.S. Pat. No. 8,456,534, which claims priority to provisional application Ser. No. 60/622,143 filed Oct. 25, 2004, the disclosures of which are incorporated herein by reference in their entirety.
The present invention relates to control systems for controlling applications related to entertainment systems, such as televisions, multimedia systems, Internet access systems and browsers, and related methods.
2. Description of the Prior Art and Related Information
A need has arisen for providing control capabilities in the living room along with the ability to control the conventional entertainment devices typically present in the living room. For example, combined PC and TV systems have been introduced which integrate the capabilities of the personal computer with the television. One such system is described in U.S. Pat. No. 5,675,390. Also, set top Internet access devices have been introduced which integrate Internet access capabilities with conventional televisions. Also, the advent of digital video recorders (DVRs), wireless networking systems for video, audio and picture transfer to TVs, and other digital devices linked to the TV has introduced many more functions to TV control, including complex display menus, introducing a need for better control of applications.
In one aspect the present invention provides a method comprising receiving image information from an imager, identifying at least one passive reference field of a user-manipulated device based on one or more colors and shapes that comprise the at least one passive reference field, the at least one passive reference field reflecting light, and determining identified reference information based on the at least one passive reference field. The method further comprises generating position information based at least in part on the identified reference information, the position information expressing a position of the user-manipulated device relative to the imager, the position information generated at least in part by mapping the position of the at least one passive reference field to one or more fixed reference points. The method further comprises controlling an application based on the generated position information.
In another aspect the present invention provides a method comprising receiving image information from an imager, identifying at least one reference field of a user-manipulated device based on one or more an angle sensitive markings that comprises the at least one reference field and determining identified reference information based on the at least one reference field. The method further comprises generating position information based at least in part on the identified reference information, the position information changing in response to movement of the user-manipulated device relative to the imager, the position information generated at least in part by detecting movement response of the angle sensitive markings. The method further comprises controlling an application based on the generated position information.
Further features and advantages of the present invention are set out in the following detailed description.
FIG. 1 is a perspective view of an improved entertainment system in accordance with the present invention in a presently preferred embodiment.
FIG. 2 is a top view of the remote controller of the present invention in a presently preferred embodiment.
FIG. 3 is a block schematic diagram illustrating control circuitry of the remote controller of the present invention.
FIG. 4 is a schematic diagram illustrating the image data captured by the remote controller of the present invention including the display screen of the entertainment system of FIG. 1.
FIG. 5 is a schematic diagram illustrating the image data after background processing, which image data corresponds to the desired display screen image data, and derived relative position information.
FIG. 6 is a flow diagram illustrating the processing of image data by the remote controller of the present invention.
FIG. 7 is a simplified schematic of the display control/input device of the system of FIG. 1.
FIG. 8 is a flow diagram illustrating the process flow of the display control/input device for converting the received position data to a cursor or other GUI multi-directional control function.
FIGS. 9A and 9B illustrate display menus having a bright boundary and/or markings for use in image isolation.
FIG. 10 is a perspective view of an alternate embodiment of the remote controller of the present invention employing a folding configuration with a text entry keyboard.
The present invention provides a remote control system and method adapted for use with an entertainment system employing a multi-directional control function such as a GUI control interface. Any such multi-directional control capability is referred to herein, for shorthand purposes only, as a GUI interface. In FIG. 1 an improved entertainment system in accordance with the present invention is illustrated in a perspective view in a presently preferred embodiment. Details of such systems beyond the novel control features described herein are known and will not be described in detail herein. For example, a PC/TV system with internet access is one example of such an entertainment system and is disclosed in the above noted '390 patent, the disclosure of which is incorporated by reference in its entirety.
Referring to FIG. 1, the entertainment system 100 includes a multi-directional remote controller 110, a display 112, which for example may be a TV monitor, a primary display control/input device 114 and a secondary display control/input device 116. Primary display control/input device 114 and secondary display control/input device 116 may comprise any of a variety of devices using a TV display for output. Primary control/input device 114 is adapted for a GUI interface control displayed on the display 112. For example, the primary input device 114 may comprise a multi-media PC such as in the above noted '390 patent or other device adapted for utilizing a multi-directional control, such as a GUI interface. Other examples of primary input device 114 include digital cable or satellite TV boxes, DVR systems, networked digital media systems adapted for media transfer from a networked PC, internet steaming media devices, digital video game players, etc. A variety of possible devices may therefore comprise primary input device 114. Furthermore the functionality of input device 114 may be incorporated in the television system 112 and is simply illustrated as a separate device for illustration of one possible configuration. Secondary input device 116 may also comprise any of a variety of known devices employed in entertainment systems and may include a DVR, cable TV box, or other digital or combined analog and digital interface device. Device 116 may incorporate a GUI type interface or a more conventional interface for TV systems adapted for, e.g. a push button LED remote control. Also, the functionality of device 116 may be incorporated along with device 114 or TV 112 and again the illustration of a separate input device is purely for illustration of a possible configuration and without limitation. Plural devices 114, 116 are shown to clarify that the control system of the present invention may control a conventional device as well as a GUI device, with an (optional) combined universal remote/multi-directional control capability in one embodiment of a controller 110 as described below.
Remote controller 110 provides a multi-directional control capability which is schematically illustrated by control of cursor 118 displayed in the monitor 112. It should be appreciated however that a variety of different multi-directional control interfaces may be employed other than a cursor such as in a typical mouse control of a PC. For example the multi-directional controller 110 may control highlighting and selection of different icons or other GUI interface layouts displayed on the screen of TV monitor 112 by device 114 and/or device 116. Also, the multi-directional controller could simply enable rapid scrolling through large channel lists such as in digital cable menus without the tedious up-down-left-right scrolling typically employed. As will be described in more detail below, remote controller 110 employs a freely movable multi-directional motion based control similar to a mouse control of a PC but without being limited to use on a flat surface.
Referring to FIG. 2, the remote controller 110 is illustrated in more detail in a top view. As shown, the remote controller may have a configuration similar to a typical remote control employed in an entertainment system. Alternatively, the controller 110 may have a shape more similar to a mouse type controller or other desirable ergonomic configuration adapted for use in one hand in a living room setting. The top surface of the controller housing 120 may include a number of first remote control inputs indicated generally at 122. This first set of control inputs 122 may include conventional remote control functions typically found in hand-held TV remote controls or universal remote controls adapted to control multiple entertainment devices such as TVs, VCRs, CD players, DVD players, etc. Therefore the first set of remote control inputs 122 may include the volume up and down set of controls 124, a channel up and down set of controls 126, a power button 128 and a set of numeric inputs 130. Also, a number of programmable or special purpose control buttons may be provided that are indicated generally as buttons 132. The first set of controls 122 activate a first wireless transmitter 134 which may preferably be an LED or RF transmitter configured at one end of the housing 120. As further illustrated in FIG. 2, the remote controller 110 preferably includes mouse type control buttons 136, 138 which comprise a second set of control inputs. Normally the multi-directional control will not be needed and may be disabled for power saving purposes. For this purpose a button 140 may be provided to activate the second set of inputs and the multi-correctional control capability of the controller 110. Button 140 may at the same time transmit a control signal to the control input device 114 to display a suitable menu adapted for multi-directional control on the display screen 112. Although one button 140 is shown several menu buttons may be provided which enable display of the appropriate menu and at the same time enable the multi-directional control capability. For example, activating a channel button would activate display of a channel list menu and simultaneously enable the multi-directional control for rapid scrolling through the channels. Alternatively, the multi-directional control may only be active while button 140 is held depressed. Activation of the button 140 may also deactivate some or all of the first set of remote control inputs 122 so that these are not inadvertently activated while the user operates the mouse type control buttons 136 and 138. Alternatively a movable cover may be provided over the first set of inputs 122 to cover these while the multi-directional control function is enabled; for example a sliding type cover may be provided for this purpose or a hinged section with inputs 112 on the inner section or sections as shown by the dashed line. Alternatively, in some applications the remote control inputs 122 may not be needed; for example in an application where the controller 110 is used in conjunction with a separate remote or with a keyboard having control functions and the controller 110 is used solely as a multi-directional input device, some or all of the controls 122 may be dispensed with. Also, some or all of the functions of inputs 122 may be allocated to GUI control on the screen. Also, the buttons 136, 138 may not be activated by separate button(s) 140 but may be active all the time and operate as a conventional "action" or "select" button when operating in a non GUI control mode and operate to select and/or provide mouse button type selection when the multi-directional control mode is active. Although two buttons 136, 138 are shown similarly to a mouse a single button may be employed (or more than two, or a scroll wheel may be added, e.g. for rapid channel navigation). The control signals from the control inputs 136, 138 and the multi-directional control signals are provided to a second transmitter 142 which may also preferably comprise either a LED or RF type wireless transmitter. Alternatively, in some applications, e.g. video game control, a wired rather than wireless transmission between the controller and device 114 may be preferred. As further shown in FIG. 2, a lens assembly 144 is provided at the front of the housing to allow the capture of image data including the display screen 112 for provision to a digital camera (or imager) and image signal processing system described in more detail below.
The controller 110 may also provide various degrees of enhanced "universal control" GUI capability over various devices, such as device 116 or TV 112. For example, most such devices will have up-down-left-right controls and associated LED control pulses control the associated menu. If control of such devices using controller 110 is employed, the detected motion of the controller 110 (described in detail below) can be converted to a high speed series of up, down, left or right LED pulses coded for the particular device being controlled. In this way more convenient navigation of conventional menus can be provided by controller 110. Alternatively, device 114 may include an LED "blaster" or direct connection to other devices (e.g. via a USB port) to provide such universal control.
Referring to FIG. 3, a block schematic diagram is illustrated showing the circuitry of the remote controller. As shown in FIG. 3, the controller circuitry includes an imager 150 which receives light captured by lens 144. Imager 150 may comprise a suitable commercially available digital imager, for example commercially available CMOS imagers providing relatively high-quality digital images are available at relatively low cost and may be advantageously employed for imager 150. The output of imager 150 will be image data corresponding to the pixels in the field of view of the imager 150, which field of view is suitably chosen by lens 144 to encompass the area in front of the controller including the display screen 112 shown in FIG. 1. The pixel data output from imager 150 is provided to a signal processor 152 which may be a suitably programmed DSP programmed in a manner to provide the image processing functions described in more detail below. The output of the DSP 152 will be data preferably corresponding to the position off set of the image of screen 112 shown in FIG. 1 from the image axis of the optics of remote controller 110. Alternatively, the data may correspond to changes in image position between frames. This position data is provided to microprocessor 154 which controls first transmitter 134 to transmit the position data to the output control device 114 (or 116) shown in FIG. 1. Microprocessor 154 will also receive inputs from switches 136 and 138 corresponding to the multi-directional control buttons shown in FIG. 2. These will also be provided to first transmitter 134 and provided to control/input device 114 (or 116) for control of the GUI functions of the display 112. The microprocessor 154 also receives inputs from activation of keys 122 (shown in FIG. 2) provided from key detect circuit 156. This key activation data is provided by microprocessor 154 to second transmitter 142 and is transmitted to the appropriate input device 114, 116 or to TV 112. Two transmitters 134 and 142 may be advantageously employed were the control signals from switches 122 provide a conventional LED type control signal which may be used for standard remote control functions in components in the entertainment system illustrated in FIG. 1. Transmitter 134 in turn may be optimized to transmit the position information and is preferably insensitive to the orientation of the control relative to the input device (114) containing the receiving circuitry. Therefore, transmitter 134 may be a sufficiently high bandwidth RF transmitter. Alternatively, however, transmitter 134 may also be an LED type transmitter. Also a single transmitter may be employed for transmitting both types of signals under the control of microprocessor 154. Microprocessor 154 may also store codes for universal control operation. An (optional) receiver 148 may also be provided, e.g. to receive a signal from device 114 that the multi-directional control menu has exited allowing controller 110 to disable the imager and DSP for power saving. Alternatively, as noted above, the multi-directional control may only be active while a button is held depressed. Alternatively, a timer may put the imager and DSP in a sleep mode if a certain time has elapsed without use of a GUI control button. Alternatively, the controller 110 may detect when the menu has exited, for example, by detecting absence of a suitable boundary or marker superimposed on the menu and disable the camera and DSP. Other information may also be received from device 114, e.g. to customize the control functions for different GUI interfaces. If device 114 has a networked wireless interface, such as a WiFi interface, controller 110 may also employ this protocol and be networked with device 114. Microprocessor 154 also receives as an input the control signal from switch 140 which, as noted above, may conveniently activate a menu or other interface signaling activation of the multi-direction controller function and a GUI interface and optionally deactivate the control inputs 122. Although the microprocessor 154 and DSP 152 are shown as separate processors in FIG. 3, it will be appreciated that the functionality of these two processors may be combined into a single microprocessor and the specific illustrated configuration of the circuitry in FIG. 3 is purely one example for illustrative purposes.
Next, referring to FIGS. 4-6 the image processing implemented by DSP 152 in FIG. 3 will be described in more detail. First of all, referring to FIG. 6 the first stage in the image processing is to capture a frame of image data as illustrated at 300. In FIG. 4 the image data captured by imager 150 is illustrated. As shown, the field of view 200 includes image data (pixels) 202 corresponding to the display screen 112 shown in FIG. 1 as well as background image data 203. The image data 202 for the display screen has several characteristics which distinguish it from the background and which allow it to be reliably detected by the image processing software. These characteristics include the following: the image data 202 from the display screen will be brighter than the background; the boundary of the image data 202 of the display screen will have straight edges; the image 202 will have a rectangular shape; and the image 202 will have a substantial size in comparison to other objects in the total field of view 200. These characteristics may be employed to eliminate the irrelevant background images and clearly discern the image 202. Also, the menu or other GUI displayed on the screen may be supplanted with specific features adapted for detection, as discussed below.
Next, referring to FIG. 6, at 302, the DSP image processing proceeds to eliminate background image data and isolate the image data for screen image 202. This processing employs some or all of the above noted unique characteristics of the image 202 to eliminate the background image data. In particular, as shown in FIG. 4 by the shaded area, a majority of the background image data 203 will have a brightness substantially less than image data 202 and this portion of the background can be rejected by rejecting the pixel data below a reference brightness threshold. The remaining groups of image data will correspond to relatively bright objects which may occur in the field of view, illustrated for exemplary purposes in FIG. 4 by image data 204, 206. For example, such image data may correspond to a bright object such as a lamp's image data 204. Also, reflected image data 206, for example corresponding to a reflection off of a coffee table or other reflective surface in the field of view may be present. Image data 204 and 206 may be readily eliminated by using additional characteristics of the desired data 202. For example, the undesired image data will in general not have straight edges and not be rectangular in shape and therefore may be readily eliminated by the signal processing. This will employ edge or boundary detection which may be easily performed since the surrounding pixel data has been eliminated by the background processing described above and a simple comparison of pixel values will derive the boundaries of the screen image 202. Also, reflections of the display screen itself may be eliminated by doing a comparison of the brightness of the two images and selecting the brighter of the two objects. Furthermore, the reflections may be substantially eliminated from the image data by employing a polarized filter in the lens assembly 144. If background image data survives which satisfies these characteristics, a comparison between image data 202 and the surviving image data in the background can be made to determine the relative size of the two objects and the smaller object eliminated.
Additionally, since the image displayed in the display screen image 202 is under the control of the input/control device 114 the image 202 may be provided with a distinct characteristic to enable its ready detection against the background image data in the field of view 200. This display will typically correspond to a unique menu or other GUI interface which can be provided with a bright boundary, or distinctive bright readily identified position markers, e.g. in the four corners of the image 202. This may be superimposed on a standard browser or menu, for example. One simple example is illustrated in FIG. 9A which illustrates a display menu having a bright background 340. Alternatively a bright boundary 352 may be provided or markings 354. Four corner markings 354 are shown which may be aligned with image center and which may be relatively small bright markers. Also these may be combined with boundary 352 with a characteristic contrast. Also, a distinctive color or combination of colors may be employed for boundary 352, markers 354 or both. As another example a logo or name of the device manufacturer (e.g. DVR manufacturer) or service provider (e.g. cable company) may be used as a unique marker 354. If the logo or name includes text standard OCR processing may be used to match the detected combination of letters to a stored template for reliable marker detection. Another example of a menu layout is shown in FIG. 9B. It will be appreciated that a variety of different suitable boundaries and/or markings may be employed to help in distinguishing the image data for the screen 112 from background image data. Such characteristics of the display may be combined with or substituted for the above noted characteristics for detection of image data 202. For example, if discrete markers 354 are employed their unique characteristics may be substituted for the rectangular shape and straight boundaries of the screen image described above to distinguish from background. As another example, if color of the boundary or marker is used, initial color discrimination may be used to reject the background.
In the unlikely event that the image processing locks onto an incorrect object a simple reset may be provided, e.g. using button 140 or some other manually activated input, and upon reset activation a further characteristic may be employed, namely position relative to the center of the field of view with objects far away from the center being rejected. This allows the user to reset the image tracking system, for example if it inadvertently locks onto a window in a room, after pointing the controller at the display screen and hitting a reset button.
After the above noted processing the remaining image data corresponds to the desired display screen image data 202, as generally illustrated in FIG. 5. The processing flow then proceeds to derive the center of the display screen image from this remaining image data at processing step 304, illustrated in FIG. 6. This may employ the rectangular image boundaries if these have been used at processing 302 or symmetric markings 354 if these are used. The process flow next proceeds to derive the relative position of the center of the screen image 208 to the center 210 of the field of view 200 (and the center of the optical axis of the controller lens assembly). As shown in FIG. 5, this offset information may be readily calculated from the image center pixel information derived previously and offset values X,Y may be derived as shown. This relative position data is transmitted to the input/control device 114 as shown at 306 in FIG. 6. It should be appreciated that other offset position references may be employed than image center. For example, top (and/or bottom) and side boundary offset values may be derived and transmitted to the device 114 as position information. Alternatively, marker position offset(s) from the imager optical axis may be determined and transmitted as position information. Adjustment for tilt may be provided during this processing, for example, by rotating the image data about the center of the imager field of view until the edges of the screen or marker correctly align with the edges of the pixel array before determining offsets. Alternatively, purely image feature motion detection may be used for the multi-directional control, without employing the relative position offset of the imager axis to the detected image feature. Instead changes in the position of the detected image feature between frames may be used to provide motion control. The position information transmitted at 306 may then be just the change in image position from a prior frame. This approach may also provide correction for tilt, for example, by rotating the image feature detected at each frame about the imager center to match the prior frame before determining the change in position. However, while the approach using imager axis offset information allows either pointing position based or motion based control, this approach only allows the latter.
In some applications with sufficiently high data rate transmissions provided, some of the above image processing may be performed in device 114 rather than in the controller and relatively large amounts of raw image data transferred. This may reduce the processor size on the controller and reduce battery drain. Alternatively, it should also be appreciated that some or all of the processing described below as performed in device 114 may be performed on board the controller 110.
Next, referring to FIGS. 7 and 8 the control processing using the received position data, provided by the input device 114, is shown. More particularly, in FIG. 7 a simplified schematic of the input/control device 114 is shown and in FIG. 8 a process flow is illustrated for the translation of the position data into cursor control of a GUI interface on the display screen 112, shown in FIG. 1. As shown in FIG. 7 the input device 114 will include a receiver 320 for receiving the position data as well as, optionally, a second receiver 322 for receiving the remote-control input signals from the control inputs 122 on the remote-control. Also, receiver 322 may be in device 116. The receivers 320, 322 are coupled to suitable demodulation and amplification circuits 324, 326, respectively which in turn provide the received data to a microprocessor 328. A transmitter 325 and modulator 327 may also be provided to communicate with the controller 110 or a networked wireless device. Microprocessor 328 will perform a number of functions which will depend on the particular device and will include additional functional blocks 330 and 332 for providing control of a GUI interface based on received position data from the controller in functional block 330 and optionally additional remote-control functions from the other inputs 122 in block 332. Although these functional blocks are illustrated as part of the system microprocessor 328 it will be appreciated they may be also provided as separate circuits or separately programmed microprocessors dedicated to the noted functions.
Referring to FIG. 8, a simplified process flow for converting the received position data to a multi-directional control function is illustrated. As shown at 350, the process flow begins when a GUI or other multi-directional control mode is entered and the appropriate display will be provided on the display screen 112. As noted above, this display screen may preferably have a bright background or may include additional bright boundaries or other characteristic markings which may assist in the accurate identification of the screen image data from background image data. Two simple examples of such a menu screen are shown in FIGS. 9A and 9B, discussed above. A number of GUI icons 356 are also illustrated in FIG. 9A along with pointer 118. In FIG. 9B a scroll bar 353 is shown, e.g. for rapid channel selection. If the pointer control function is used in web navigation, a bright boundary or marker may be superimposed on the web pages displayed. Next as shown at 360 in FIG. 8 the process flow activated by entry into the multi-directional control mode operates to receive the position information from the controller 110 provided from receiver 320. At 370 the received position information is then processed and translated to cursor position information. Converting the position information to cursor position control information at 370 may employ a variety of different functions depending on the particular application and entertainment system configuration and intended use. In general, this translation operation will provide a mapping between the received position information and cursor position based on a sensitivity which may be user adjustable. In particular, the user may choose to adjust the sensitivity based on how close the screen is to the user which will affect the amount of angular motion of the controller 110 required to move the cursor a particular amount in the display screen. Also, the processing at 370 may employ as an input the aspect ratio of the screen and an aspect ratio of the detected image data 202 may be derived (by the microprocessor 154 or by the DSP 152 in controller 110 and transmitted along with the position information). These two aspect ratios may be compared to derive an angle at which the user is configured relative to the screen and this angle may be used to adjust the sensitivity of the received position to cursor map at 370. That is, when the user is directly in front of the screen movement of the controller will require the maximum angular movement to move the cursor in the horizontal direction and the control sensitivity of the map of position information to cursor control at 370 may be made more sensitive. Conversely, when the angle of the user relative to the screen is greater a smaller movement of the controller will cover the range of cursor movement in the horizontal direction and a less sensitive mapping at 370 may be employed. In this way the control function will have the same feel irrespective of position of the user. Similarly, compensation processing may be provided at 370 for tilt of the controller field of view relative to the screen as well as vertical angle adjustment. Next at 380 the process flow proceeds to compare the change in cursor positions (and/or change in position data) between different frames of image data to smooth out the cursor control. This processing at 380 may be employed to reject jitter by averaging motion over several frames of image data or by rejecting sudden irregular changes. Such jitter rejection processing may also be adjustable by the user. Finally at 390 the cursor position (or other position indicated) on the display is updated and the modified GUI screen is displayed.
In another implementation, the cursor itself may be chosen as the unique marker displayed on the menu (or other GUI image displayed on the screen by device 114) and its position offset from the imager's center axis detected. The device 114 will then perform processing to move the displayed cursor to eliminate the offset. In other words the cursor will be moved to where the controller imager axis is pointing. Alternatively, as noted above, instead of controlling a cursor the multi-directional control may control highlighting of menu items or in a video game application control movement in the video game.
As noted above the use of detected image feature/imager axis position offset for motion control of the GUI screen allows either remote pointing position or motion control to be employed. That is the GUI cursor or highlighting control will either be based on where the remote is pointing on the screen or will simply change based on the change in the imager axis offset position from frame to frame. The latter control is of a familiar type to computer users familiar with mouse control and may be preferred where the primary use of the remote control is in a PC/TV type multi-media system for controlling the PC type functions or in an internet access enabled TV based multi-media system. The pointing position based control in turn may be preferred for TV menu control, especially for users unfamiliar with mouse control of a PC, or for video game control. Also, the preferences may change between modes of the multi-media system. For example in a multi-media system with internet access, digital TV, and video game capability it may be desirable to switch modes from pointing position to motion control depending on what aspect of the system is in use. Also, it may be desirable to allow users to choose their preferred control mode. Therefore, in another aspect of the invention one of the buttons illustrated on the remote control 110 may be a mode selection button and initiate a corresponding processing mode in the DSP 152 or the control 100 or in the device 114 in microprocessor 328 to optionally provide pointing position or motion based control.
In an embodiment where the controller operates with more conventional devices by providing suitable coded high speed left-right-up-down control pulses (e.g. LED pulses) to simulate the use of left-right-up-down buttons processing may be provided to convert the detected position reference information to control pulses transmitted to device 116 or TV 112. This processing may be performed on controller 110. Alternatively, if the universal control is provided via a device 114 specially adapted for universal control in conjunction with the controller 110 then this processing may be provided in device 114 and the other device(s) (e.g. 116) controlled via an LED blaster or direct connection such as a USB port.
Referring to FIG. 10, an optional or alternate embodiment of the system of FIG. 1 is illustrated employing a combined multi-directional controller and wireless keyboard 400. The multi-directional capability of controller 400 may be precisely the same as described above in relation to the controller 110 of the prior embodiments but in addition a keyboard configuration 402 for text entry may be provided. More specifically, the controller 400 may have a folding housing with the keyboard 402 configured in the interior of the controller 400. This embodiment may incorporate the teachings of U.S. Pat. No. 6,094,156, the disclosure of which is incorporated herein by reference in its entirety, and accordingly the details of such an alternate embodiment need not be described in further detail since the necessary modifications to controller 110 are apparent from the aforementioned patent. It should be appreciated that such an embodiment may also optionally include inputs 122 described above and provide the capabilities of three distinct devices normally provided by separate controllers, namely a universal remote control, multi-directional control and keyboard text entry control normally found in three separate remote control devices.
Other implementations may be provided that may not be preferred in many applications but may be suitable in others. For example, a unique marking (such as 354 in FIG. 9) may be provided on the front of display 112 or device 114, e.g. by a pattern of LEDs or illuminated distinctive logo. This could be imaged instead of screen 112 and used to detect controller motion for pointer control as above. Alternatively, in such an embodiment, a distinctive angle sensitive marking may be provided and motion detected by detection of the changes in brightness of the detected markings. For example, a light source covered with a transparent diffractive pattern, e.g. a hologram or diffractive grating may be used. In yet another embodiment a unique marking may be provided on controller 110 and imaged by device 114 to detect motion. In this embodiment, the imager 150 is configured in device 114 and the process flow of FIGS. 5-6 is performed in processor 128. All such embodiments are implied herein.
It will be appreciated that the before mentioned embodiments of the invention may provide a variety of different capabilities depending on the particular application and system. In one aspect, the controller 110 may be operated as a primary control for a GUI based interface for a TV or other display based entertainment or information system with the functionality normally provided by a universal remote provided by the GUI interface and controlled by the multi-directional controller and the additional controls 122 shown in FIG. 1 need not be provided without loss of functionality. Alternatively, the controls 122 may be provided with the remote controller 110 providing a combined multi-directional controller and universal remote in one simple handheld configuration. Either embodiment may advantageously be used in a TV/cable/DVR environment or in a TV/PC enhanced environment and provide enhanced functionality. One example of the advantages of such control in a digital cable TV environment is apparent from FIG. 9B. As shown, with scroll bars and selected speed scrolling (by positioning the pointer below the listing a desired amount to control scrolling speed) the navigation of hundreds of channels may be greatly simplified. Also, in another configuration the multi-directional control capabilities may be combined with a separate wireless keyboard to provide enhanced functionality for PC/TV type systems or Internet access systems. Alternatively, the controller may provide multi-directional control and text entry in the embodiment of FIG. 10. Also, the combination of universal remote, multi-directional controller and text based input may all be provided in a single compact handheld configuration in the embodiment of FIG. 10. In another application the device 114 may be a video game device or incorporate video game capabilities and the remote control may provide a control over the video game replacing a joystick or other multi-directional controller.
Also, the indentified distinctive marker may be used for other types of application control. For example, as noted above, the marker may be a logo or name. Once this marker is identified various additional control applications are possible. For example, identifying a logo or name may allow information corresponding to the logo or name to be accessed via the internet. As another example, if the marker is a cable company identifier or broadcast company identifier the marker identifier may be used to synchronize the displayed content to another device, or the device 110 if it includes a display (for example, if device 110 is a smart phone or tablet.)
Therefore, it will be appreciated that the marker identification described above may be employed for additional applications other than cursor control or selection of applications from a GUI interface. These applications may be controlled in parallel with or independently of position tracking using the image data.
As one specific example, if device 110 is a smart phone or tablet with the above described marker recognition capability, the user may control operation of an application whereby the device 110 identifies a service provider identifying marker or logo to identify a channel and time. This information may then allow continued access to the same content on the device 110 at the same point in the program via a wireless connection on device 110. As another specific example, the user may point device 110 at a product with a distinctive logo being displayed in a program and identify the logo and product to access additional information, either to be displayed on device 110 or on display 112 via an internet access application. Also, the control of an application independently of tracking need not be performed in real time so some or all of the control and image identification processing may be performed at a separate location employing the image information from the device 110; for example, employing a logo or product image database at a remote location accessed via an internet connection. Such a remote database may also include product placement and program information. In this case the logo identification of a channel and a time may enable a remote application which identifies a product and product information corresponding to a program time slot which product information is then provided back to the user. Also, the position of an object in the image relative to a fixed logo position in the image may be known and used for product identification. Therefore, applications controlled may generally include product object identification combined with provider logo information.
As another example, the tracking control of a cursor will include the coordinates of the cursor as described above. The coordinates of a product placed in the image may also be known in advance and either provided in meta data sent along with the image to system 100 or kept in a remote product placement database. Then when the user points the cursor to a product in the image and selects a product identification and information application the coordinates may simply be provided to the product database or the meta data coordinates may be identified and used to access an encoded product site via the internet, e.g., by associating the transmitted coordinate meta data with a URL. This approach to product identification may employ other pointer based controllers than image based pointer control as described above, for example gyroscopic or accelerometer controllers, and use of such other systems in this type of product identification application are within the scope of this aspect of the invention. Nonetheless an imager based approach has the advantage of direct pointing control which is more intuitive for a user. Also, as noted above the imager may be based at or near the display and pointing coordinates identified in that manner, whether by imaging and detecting device 110 or gesture detection. As another example, a user may capture the image on a device 110 such as a smart phone or tablet and identify product coordinates by a touch interface or stylus and simply touching or encircling the product on the captured image. Such alternate approaches are similarly within the scope of this aspect of the invention associating display image coordinates, or range of coordinates, with product placement meta data, identifying a product location in an image by a user and employing this for retrieval of product information.
It will be appreciated by those skilled in the art that the foregoing is merely an illustration of the present invention in currently preferred implementations. A wide variety of modifications to the illustrated embodiments are possible while remaining within the scope of the present convention. Therefore, the above description should not be viewed as limiting but merely exemplary in nature.
receiving image information from an imager;
identifying at least one passive reference field of a user-manipulated device based on one or more colors and shapes that comprise the at least one passive reference field, the at least one passive reference field reflecting light in a diffractive pattern;
determining identified reference information based on the at least one passive reference field;
generating position information based at least in part on the identified reference information, the position information expressing a position of the user-manipulated device relative to the imager, the position information generated at least in part by mapping the position of the at least one passive reference field to one or more fixed reference points; and
controlling an application based on the generated position information.
2. The method of claim 1, wherein the imager is configured to image the user-manipulated device when the user-manipulated device is pointed toward a display device.
3. The method of claim 1, wherein controlling an application based on the generated position information comprises controlling a display.
4. The method of claim 1, wherein controlling an application based on the generated position information comprises controlling a pointer on the display using the position information.
5. The method of claim 1, wherein controlling an application based on the generated position information comprises controlling a game on the display using the position information.
6. The method of claim 1, wherein the one or more fixed reference points are defined by reference to the imager field of view.
7. The method of claim 1, wherein the one or more fixed reference points comprise the center of the imager field of view.
8. The method of claim 1, wherein the user-manipulated device comprises a game controller.
9. The method of claim 1, wherein the user-manipulated device comprises a remote control.
identifying at least one reference field of a user-manipulated device based on one or more diffractive patterns that comprises the at least one reference field;
determining identified reference information based on the at least one reference field;
generating position information based at least in part on the identified reference information, the position information changing in response to movement of the user-manipulated device relative to the imager, the position information generated at least in part by detecting movement response of the one or more diffractive patterns; and
11. The method of claim 10, wherein the reference field comprises a diffractive grating.
12. The method of claim 10, wherein the reference field comprises a holographic marking.
13. The method of claim 10, wherein controlling an application based on the generated position information comprises controlling a display.
14. The method of claim 10, wherein controlling an application based on the generated position information comprises controlling a pointer on the display using the position information.
15. The method of claim 10, wherein controlling an application based on the generated position information comprises controlling a game on the display using the position information.
16. An entertainment control system, comprising:
an imager;
a user movable object having one or more diffractive patterns that comprise at least one reference field;
a control device coupled to receive image information from the imager, the control device including processing circuitry for identifying the at least one reference field of the movable object based on the one or more diffractive patterns that comprises the at least one reference field; determining identified reference information based on the at least one reference field; generating position information based at least in part on the identified reference information, the position information changing in response to movement of the object relative to the imager, the position information generated at least in part by detecting movement response of the one or more diffractive patterns; and controlling an application based on the generated position information.
17. An entertainment control system as set out in claim 16, wherein the reference field comprises a diffractive grating.
18. An entertainment control system as set out in claim 16, wherein the reference field comprises a holographic marking.
19. An entertainment control system as set out in claim 16, wherein the application controls a display in response to movement of the object.
20. An entertainment control system as set out in claim 19, wherein the application controls a game on a display using the object movement information.
US14/491,863 2004-10-25 2014-09-19 Generating position information employing an imager Active US9405431B2 (en)
US11/255,647 US8456534B2 (en) 2004-10-25 2005-10-21 Multi-directional remote control system and method
US13/908,900 US8842186B2 (en) 2004-10-25 2013-06-03 Control system and method employing identification of a displayed image
US14/491,863 US9405431B2 (en) 2004-10-25 2014-09-19 Generating position information employing an imager
US15/190,140 US9965027B2 (en) 2004-10-25 2016-06-22 Control system employing identification of a displayed image
US15/973,349 US10152115B2 (en) 2004-10-25 2018-05-07 Generating position information employing an imager
US13/908,900 Continuation US8842186B2 (en) 2004-10-25 2013-06-03 Control system and method employing identification of a displayed image
US15/190,140 Continuation US9965027B2 (en) 2004-10-25 2016-06-22 Control system employing identification of a displayed image
US13/908,900 Active US8842186B2 (en) 2004-10-25 2013-06-03 Control system and method employing identification of a displayed image
US14/491,863 Active US9405431B2 (en) 2004-10-25 2014-09-19 Generating position information employing an imager
US15/190,140 Active 2025-10-27 US9965027B2 (en) 2004-10-25 2016-06-22 Control system employing identification of a displayed image
US15/973,349 Active US10152115B2 (en) 2004-10-25 2018-05-07 Generating position information employing an imager
US9213414B1 (en) 2009-11-13 2015-12-15 Ezero Technologies Llc Keyboard with integrated touch control
KR101789683B1 (en) * 2011-06-13 2017-11-20 삼성전자주식회사 Display apparatus and Method for controlling display apparatus and remote controller
CN202818524U (en) * 2011-10-11 2013-03-20 Lg电子株式会社 Multimedia device and remote controller
FR3018983B1 (en) * 2014-03-20 2016-05-06 Viaccess Sa Method and device for identifying a content being viewed on a screen
CN104270664B (en) * 2014-10-29 2017-09-05 上海联彤网络通讯技术有限公司 Remote control light pen, the system and method of the intelligent platform to achieve control input
KR101664038B1 (en) * 2014-12-09 2016-10-10 현대자동차 주식회사 Concentration manipulation system for vehicle
CN105898516A (en) * 2015-12-15 2016-08-24 乐视网信息技术(北京)股份有限公司 TV focus presenting method and device
JPH0234633A (en) 1988-06-15 1990-02-05 Rhone Poulenc Chim Crystallized semi-aromatic polyamide having boiling point lower than 290°c
US5181181A (en) 1990-09-27 1993-01-19 Triton Technologies, Inc. Computer apparatus input device for three-dimensional information
US5227985A (en) 1991-08-19 1993-07-13 University Of Maryland Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monitored object
JPH06250285A (en) * 1993-02-24 1994-09-09 Olympus Optical Co Ltd Camera provided with remote control device
US5461478A (en) 1991-12-26 1995-10-24 Fanuc Ltd. Method and apparatus for measuring three-dimensional position and orientation of an object using light projection
US5617312A (en) 1993-11-19 1997-04-01 Hitachi, Ltd. Computer system that enters control information by means of video camera
US5686942A (en) 1994-12-01 1997-11-11 National Semiconductor Corporation Remote computer input system which detects point source on operator
US5774357A (en) * 1991-12-23 1998-06-30 Hoffberg; Steven M. Human factored interface incorporating adaptive pattern recognition based controller apparatus
US5784282A (en) 1993-06-11 1998-07-21 Bertin & Cie Method and apparatus for identifying the position in three dimensions of a movable object such as a sensor or a tool carried by a robot
US5795306A (en) 1994-03-10 1998-08-18 Mitsubishi Denki Kabushiki Kaisha Bodily state detection apparatus
US5818424A (en) 1995-10-19 1998-10-06 International Business Machines Corporation Rod shaped device and data acquisition apparatus for determining the position and orientation of an object in space
US5856844A (en) 1995-09-21 1999-01-05 Omniplanar, Inc. Method and apparatus for determining position and orientation
US5926168A (en) 1994-09-30 1999-07-20 Fan; Nong-Qiang Remote pointers for interactive televisions
JP2000181601A (en) 1998-12-18 2000-06-30 Fujitsu General Ltd Information display system
JP2001236181A (en) 2000-02-22 2001-08-31 Fuji Electric Co Ltd Pointing device
US20020039111A1 (en) 2000-06-27 2002-04-04 James Gips Automated visual tracking for computer access
US6377249B1 (en) * 1997-11-12 2002-04-23 Excel Tech Electronic light pen system
US6417836B1 (en) 1999-08-02 2002-07-09 Lucent Technologies Inc. Computer input device having six degrees of freedom for controlling movement of a three-dimensional object
US20020126090A1 (en) 2001-01-18 2002-09-12 International Business Machines Corporation Navigating and selecting a portion of a screen by utilizing a state of an object as viewed by a camera
US6456728B1 (en) 1998-01-27 2002-09-24 Kabushiki Kaisha Toshiba Object detection apparatus, motion control apparatus and pattern recognition apparatus
US6522312B2 (en) 1997-09-01 2003-02-18 Canon Kabushiki Kaisha Apparatus for presenting mixed reality shared among operators
US20030052859A1 (en) 2001-07-05 2003-03-20 Finley Michael Cain Laser and digital camera computer pointer device system
US6538645B1 (en) 2000-10-26 2003-03-25 Sunplus Technology Co., Ltd. Computer input system utilizing a camera to sense point source
KR20040027561A (en) 2004-02-12 2004-04-01 학교법인 한국정보통신학원 A TV system with a camera-based pointing device, and an acting method thereof
US6720949B1 (en) 1997-08-22 2004-04-13 Timothy R. Pryor Man machine interfaces and applications
US6795068B1 (en) 2000-07-21 2004-09-21 Sony Computer Entertainment Inc. Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program
US6863609B2 (en) 2000-08-11 2005-03-08 Konami Corporation Method for controlling movement of viewing point of simulated camera in 3D video game, and 3D video game machine
US20050062719A1 (en) 1999-11-05 2005-03-24 Microsoft Corporation Method and apparatus for computer input using six degrees of freedom
US20050068198A1 (en) 2001-12-19 2005-03-31 Roberto Brega Method and device for tracking objects, especially for monitoring traffic
US20050110781A1 (en) * 2003-11-25 2005-05-26 Geaghan Bernard O. Light emitting stylus and user input device using same
US6921332B2 (en) 2000-11-16 2005-07-26 Konami Corporation Match-style 3D video game device and controller therefor
US6987504B2 (en) 1993-07-16 2006-01-17 Immersion Corporation Interface device for sensing position and orientation and outputting force to a user
US20060050052A1 (en) 2002-11-20 2006-03-09 Mekenkamp Gerhardus E User interface system based on pointing device
JP2006079249A (en) 2004-09-08 2006-03-23 Matsushita Electric Ind Co Ltd Communication terminal
US20060287083A1 (en) 2005-05-23 2006-12-21 Microsoft Corporation Camera based orientation for mobile devices
JP2007005983A (en) 2005-06-22 2007-01-11 Fujifilm Holdings Corp Photographing apparatus and image storage control method and program thereof
US20070058047A1 (en) 2004-10-25 2007-03-15 Henty David L Multi-directional remote control system and method
US20070257884A1 (en) 2006-05-08 2007-11-08 Nintendo Co., Ltd. Game program and game system
US20070273464A1 (en) 2005-11-25 2007-11-29 Hitachi Plant Technologies, Ltd. Alignment Method and System for Electromagnet in High-Energy Accelerator
US20080024435A1 (en) 2006-07-25 2008-01-31 Nintendo Co., Ltd. Information processing device and storage medium storing information processing program
US20080100825A1 (en) 2006-09-28 2008-05-01 Sony Computer Entertainment America Inc. Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen
US20080117167A1 (en) 2006-11-17 2008-05-22 Nintendo Co., Ltd Storage medium having stored thereon program for adjusting pointing device, and pointing device
US20090009596A1 (en) 2006-02-13 2009-01-08 Max-Viz, Inc System for and method of synchronous acquisition of pulsed source light in performance of monitoring aircraft flight operation
US7489299B2 (en) 2003-10-23 2009-02-10 Hillcrest Laboratories, Inc. User interface devices and methods employing accelerometers
US7711146B2 (en) 2006-03-09 2010-05-04 General Electric Company Method and system for performing image re-identification
US7920718B2 (en) 2002-09-05 2011-04-05 Cognex Corporation Multi-zone passageway monitoring system and method
US8013838B2 (en) 2006-06-30 2011-09-06 Microsoft Corporation Generating position information using a video camera
JPH0934633A (en) 1995-07-17 1997-02-07 Sanyo Electric Co Ltd Space mouse and space mouse system
US5786804A (en) * 1995-10-06 1998-07-28 Hewlett-Packard Company Method and system for tracking attitude
US6791531B1 (en) * 1999-06-07 2004-09-14 Dot On, Inc. Device and method for cursor motion control calibration and object selection
US6724368B2 (en) * 2001-12-14 2004-04-20 Koninklijke Philips Electronics N.V. Remote control system and method for a television receiver
US20040027328A1 (en) * 2002-08-07 2004-02-12 Yao-Chi Yang Presentation system
"SmartNAV3 AT: How It Works," available at <<http://www.naturalpoint.com/smartnav/products/at/at-howitworks.html>> accessed on Apr. 19, 2006, 2 pages.
"SmartNAV3 AT: How It Works," available at > accessed on Apr. 19, 2006, 2 pages.
Cutler, et al., "View-based interpretation of real-time optical flow for gesture recognition," Proc. Third IEEE Conference on Face and Gesture Recognition, Nara, Japan, Apr. 1998, 6 pages.
Final Office Action for U.S. Appl. No. 13/208,003, mailed on Jul. 23, 2012, Feng-Hsiung Hsu, "Generating Position Information Using a Video Camera", 12 pgs.
Hinckley, et al., "The VideoMouse: A Camera-Based Multi-Degree-of-Freedom Input Device," UIST '99, ACM 1999, CHI Letters, vol. 1, pp. 103-112.
Jeon, et al., "Interaction Techniques in Large Display Environments using Hand-held Devices", VRST '06, ACM, 2006, pp. 4.
Murph, "Sony patents LED-infused, motion-tracking controller", retrieved on May 3, 2007, at <<http://www.engadget.com/2006/12/14/sony-patents-led-infused-motion-tracking-controller/>>, engadget, Weblogs, Inc., Dec. 14, 2006, pp. 1-6.
Murph, "Sony patents LED-infused, motion-tracking controller", retrieved on May 3, 2007, at >, engadget, Weblogs, Inc., Dec. 14, 2006, pp. 1-6.
Office Action for U.S. Appl. No. 11/774,515, mailed on Jan. 5, 2012, Feng-Hsiung Hsu, "Multi-Asis Motion-Based Remote Control", 31 pgs.
Office Action for U.S. Appl. No. 11/774,515, mailed on Jul. 21, 2011, Feng-Hsiung Hsu, "Multi-Axis Motion-Based Remote Control".
Office Action for U.S. Appl. No. 13/208,003, mailed on Dec. 22, 2011, Feng-Hsiung Hsu, "Generating Position Information Using a Video Camera", 15 pgs.
Office Action for U.S. Appl. No. 13/208,003, mailed on Jan. 11, 2013, Feng-Hsiung Hsu, "Generating Position Information Using a Video Camera", 9 pgs.
Response to Office Action dated Jul. 23, 2012 for U.S. Appl. No. 13/208,003 filed on Nov. 21, 2012, 17 pgs.
TrackIR product description, available at <<http://www.naturalpoint.com/trackir/02-products/product-how-TrackIRworks.html>> accessed on Apr. 19, 2006, 2 pages.
TrackIR product description, available at > accessed on Apr. 19, 2006, 2 pages.
US10152115B2 (en) 2018-12-11
US9176590B2 (en) 2015-11-03 Systems and methods for hand gesture control of an electronic device
US7787660B2 (en) 2010-08-31 Fingerprint detecting wireless device
US6160899A (en) 2000-12-12 Method of application menu selection and activation using image cognition
CN100334531C (en) 2007-08-29 User interface system based on pointing device
US7796116B2 (en) 2010-09-14 Electronic equipment for handheld vision based absolute pointing system
CA2663194C (en) 2017-07-04 Remote control unit for a programmable multimedia controller
JP4666808B2 (en) 2011-04-06 An image display system, an image display method, and storage medium, the program
US6501516B1 (en) 2002-12-31 Remotely controlling video display devices
CN1290320C (en) 2006-12-13 Remote control system and method for a television receiver
KR100957336B1 (en) 2010-05-12 Portable electronic device with mouse-like capabilities
US20110134112A1 (en) 2011-06-09 Mobile terminal having gesture recognition function and interface system using the same
US20110090147A1 (en) 2011-04-21 Touchless pointing device
US20030007104A1 (en) 2003-01-09 Network system
US20090102800A1 (en) 2009-04-23 Interactive input system, controller therefor and method of controlling an appliance
US9390714B2 (en) 2016-07-12 Control method using voice and gesture in multimedia device and multimedia device thereof
US6995751B2 (en) 2006-02-07 Method and apparatus for navigating an image using a touchscreen
US8089455B1 (en) 2012-01-03 Remote control with a single control button
US20040061788A1 (en) 2004-04-01 Multiple mode capture button for a digital camera
EP1837743A2 (en) 2007-09-26 Pointing input device, method, and system using image pattern
US9811240B2 (en) 2017-11-07 Operating method of image display apparatus
US20040095317A1 (en) 2004-05-20 Method and apparatus of universal remote pointing control for home entertainment system and computer
US20060197756A1 (en) 2006-09-07 Multi-mode optical pointer for interactive display system
US20100141578A1 (en) 2010-06-10 Image display control apparatus, image display apparatus, remote controller, and image display system
EP2453388A1 (en) 2012-05-16 Method for user gesture recognition in multimedia device and multimedia device thereof
US20020085097A1 (en) 2002-07-04 Computer vision-based wireless pointing system
Owner name: I-INTERACTIVE LLC, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HENTY, DAVID L.;REEL/FRAME:034428/0920


text:
Sam Bradford Excited for First Full Offseason with Vikings Offense
Team Reporter
EDEN PRAIRIE, Minn. —The chaos that was the 2016 season is long gone for Sam Bradford.
The Vikings quarterback doesn't have to endure weekly cram sessions on a team he joined just eight days before the start of the season.
Instead, Bradford is using the Vikings voluntary offseason workouts as a chance to catch his breath on all things football and start looking ahead to 2017.
"Obviously, last year was pretty unique. I've never been in that situation, and I don't think most people probably have been in that situation," Bradford said Tuesday from Winter Park. "But just to be here this offseason, be able to go through the program, be able to go through the meetings and installs and really learn this offense and what we're trying to do, it's a much better situation than showing up here eight or nine days before the first game and trying to kind of learn everything on the fly."
Bradford was acquired in a trade with Philadelphia on Sept. 3 and was almost immediately thrown into the fire.
Although he didn't play in Minnesota's season-opening win, Bradford started the final 15 games of the season and compiled a strong stat line — 395 completions in 552 attempts (for an NFL record 71.6 completion percentage) for 3,877 yards with 20 touchdowns and five interceptions.
Bradford said his comfort level increased as the season went along. He also dealt with some upheaval as Pat Shurmur took over as the Vikings offensive coordinator midway through the 2016 campaign.
"I think the later we got in the year, the better I felt with it," Bradford said. "Obviously, going through the (offensive coordinator) change that we did halfway through the season … having worked with Pat like I had before, I think that really helped me because we have a really good relationship. I felt like we were able to communicate.
"Toward the end of the year, I felt like I had a pretty good grasp of things," Bradford added.
A pair of Bradford's top targets said Tuesday that they are ecstatic to spend a full offseason with their starting quarterback.
Wide receiver Stefon Diggs had a team-high 84 catches for 903 yards with three touchdowns in 2016.
Tight end Kyle Rudolph had 83 receptions for 840 yards and a team-high seven touchdowns.
Both players said they were excited to see how the Vikings offense progresses over the next few months.
"It's great," Diggs said. "We did some good things (in 2016), saw a little spark here and there with what we could be. With the offseason and the preparation going into it, things should manifest on their own."
Added Rudolph: "For Sam, he doesn't have to come in (eight days before the season opener). He gets an offseason with us, and we get to go out and throw the ball around a little bit. You add a couple new guys, and we're trying to bring them along as quick as possible. We're just trying to lay this foundation for 2017 as an offense to help our team win more football games."
Bradford noted that while the focus is on football during offseason workouts, the time spent together also gives players a chance to bond off the field.
"I think that's a big part of this whole time we're here. Just getting everyone back together and spending time together and being in the building together," Bradford said. "It's good to get into meetings and talk ball and get back into it after having a nice break.
"But the biggest part of this is the camaraderie … building those relationships, building those bonds that you can lean on during the season," Bradford added. "Not having all of our effort put into game-planning and the task at hand, I think this time of year is great for (bonding). We can put more focus on a little more than just football."
The 2016 season was unlike any other that Bradford had been involved with. He's hoping 2017 is a little calmer.
"I learned a lot about a lot of things last year going through that," Bradford said. " I just think it makes me grateful to have the opportunity to be here at this time of year and be able to go through the offseason program and to hopefully prepare myself as you would in a normal year."

redpajama_set_name: RedPajamaCommonCrawl


Injectable wrinkle fillers can give you a more youthful look by erasing your wrinkles for a fraction of what a traditional face lift costs. Unlike Botox injections, which relax underlying muscles, wrinkle fillers can smooth out facial lines by filling the wrinkle or crease one of different substances. They can also be used as volumizers to plump up and fill out thin lips and cheeks.
How are wrinkle filler procedures done?
Dr. Greco performs wrinkle filler injections that will refill the hollows under the eyes and brows, and decrease the appearance of frown lines and skin folds around the nose and lips. Most wrinkle filler treatments take less than 20 minutes to fill lines, wrinkles and folds. The results can last 3 months to a year or more.
Injections are made with a small needle into or under the skin. Most treatments are accompanied by only mild discomfort. There may be some swelling at the injection site, which can last for a few hours, and mild bruising, which can last for a few days. Occasionally, there will be an allergic reaction that can consist of itching, prolonged redness or firmness at the injection site but this is rare.


text:
Q: How can one compute this simple infinite product?
Possible Duplicate:
Limit of a particular variety of infinite product/series

Define $$F(x) = \prod_{n=1}^\infty(1-x^n)$$ where $|x|<1$.
How can one compute $F(1/2)$? (Without an obvious polynomial expansion or brute-force calculation.)
This is sometimes called Euler's function.

A: That product is not so simple as you think. Euler proved that $$F(x) = \prod_{n=1}^\infty(1-x^n)=\sum_{-\infty}^{+\infty}(-1)^kx^{\frac{k(3k+1)}{2}}$$
This problem arises in partition number theory and is called Euler's pentagonal number theorem.

redpajama_set_name: RedPajamaStackExchange


text:
<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE tiles-definitions PUBLIC
       "-//Apache Software Foundation//DTD Tiles Configuration 2.1//EN"
       "http://tiles.apache.org/dtds/tiles-config_2_1.dtd">

<tiles-definitions>

    <definition name="kcpt.basePage" template="/pages/common/MainLayout.jsp">
        <put-attribute name="banner" value="/pages/common/Banner.jsp"/>
        <put-attribute name="stylesAndScripts" value="/pages/common/StylesAndScripts.jsp"/>
        <put-attribute name="rightSection" value="/pages/common/RightSection.jsp"/>
        <put-attribute name="footer" value="/pages/common/Footer.jsp"/>
    </definition>

    <definition name="kcpt.basePageNoRightSection" template="/pages/common/MainLayout.jsp">
        <put-attribute name="banner" value="/pages/common/Banner.jsp"/>
        <put-attribute name="stylesAndScripts" value="/pages/common/StylesAndScripts.jsp"/>
        <put-attribute name="rightSection" value="/pages/common/Filler.jsp"/>
        <put-attribute name="footer" value="/pages/common/Footer.jsp"/>
    </definition>

    <definition name="kcpt.baseLoginPage" template="/pages/common/CenterLayout.jsp">
        <put-attribute name="banner" value="/pages/common/Banner.jsp"/>
        <put-attribute name="stylesAndScripts" value="/pages/common/StylesAndScripts.jsp"/>
        <put-attribute name="footer" value="/pages/common/Footer.jsp"/>
    </definition>

    <definition name="kcpt.authenticate" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/login/include/authenticate.jsp"/>
    </definition>

    <definition name="kcpt.error" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/common/ErrorExceptionInclude.jsp"/>
    </definition>

    <definition name="kcpt.shibError" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/login/include/shibError.jsp"/>
    </definition>

    <definition name="kcpt.createAccount" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/login/include/createAccount.jsp"/>
    </definition>

    <definition name="kcpt.getLoginPassword" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/login/include/getLoginPassword.jsp"/>
    </definition>

    <definition name="kcpt.loginContactUs" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/plan/include/contactUs.jsp"/>
    </definition>

    <definition name="kcpt.contactUs" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/contactUs.jsp"/>
    </definition>

    <definition name="kcpt.annualReviews" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/annualReviews.jsp"/>
    </definition>

    <definition name="kcpt.toKnowItems" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/toKnowItems.jsp"/>
    </definition>

    <definition name="kcpt.faqs" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/faqs.jsp"/>
    </definition>

    <definition name="kcpt.references" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/references.jsp"/>
    </definition>

    <definition name="kcpt.sessionRole" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/sessionRole.jsp"/>
    </definition>

    <definition name="kcpt.myProfile" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/myProfile.jsp"/>
    </definition>

    <definition name="kcpt.approveNewUsers" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/approveNewUsers.jsp"/>
    </definition>

    <definition name="kcpt.begin" template="/pages/plan/include/begin.jsp">
        <put-attribute name="stylesAndScripts" value="/pages/common/StylesAndScripts.jsp"/>
        <put-attribute name="footer" value="/pages/common/Footer.jsp"/>
    </definition>

    <definition name="kcpt.welcome" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/welcome.jsp"/>
    </definition>

    <definition name="kcpt.getBasePlans" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/getBasePlans.jsp"/>
    </definition>

    <definition name="kcpt.getPlans" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/getPlans.jsp"/>
    </definition>

    <definition name="kcpt.editPlan" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/editPlan.jsp"/>
    </definition>

    <definition name="kcpt.printMenu" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/printMenu.jsp"/>
    </definition>

    <definition name="kcpt.getSystemDomainDocuments" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/getSystemDomainDocuments.jsp"/>
    </definition>

    <definition name="kcpt.getPlanUsers" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/getPlanUsers.jsp"/>
    </definition>

    <definition name="kcpt.getPlanReview" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/getPlanReview.jsp"/>
    </definition>

    <definition name="kcpt.planReviewComplete" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/planReviewComplete.jsp"/>
    </definition>

    <definition name="kcpt.planReviewAnnual" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/planReviewAnnual.jsp"/>
    </definition>

    <definition name="kcpt.basePlanPage" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/basePlanContent.jsp"/>
        <put-attribute name="planNavigation" value="/pages/common/NavigationBar.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.planHome" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/planHome.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.baseCriticalFunctionPage" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/baseCriticalFunctionContent.jsp" cascade="true"/>
        <put-attribute name="criticalFunctionNavigation" value="/pages/common/CriticalFunctionNavigationBar.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.baseInformationTechnologyPage" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/baseInformationTechnologyContent.jsp" cascade="true"/>
        <put-attribute name="informationTechnologyNavigation" value="/pages/common/InformationTechnologyNavigationBar.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.baseInstructionPage" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/baseInstructionContent.jsp" cascade="true"/>
        <put-attribute name="instructionNavigation" value="/pages/common/InstructionNavigationBar.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.baseKeyResourcesPage" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/baseKeyResourcesContent.jsp" cascade="true"/>
        <put-attribute name="keyResourcesNavigation" value="/pages/common/KeyResourcesNavigationBar.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getDepartmentInfo" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getDepartmentInfo.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionExamples" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/plan/include/getCriticalFunctionExamples.jsp"/>
    </definition>

    <definition name="kcpt.getCriticalFunctions" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getCriticalFunctions.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionActionItem" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionActionItem.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionActionItems" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionActionItems.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionConsequences" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionConsequences.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionCopingMethod" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionCopingMethod.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionDependencies" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionDependencies.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionDescription" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionDescription.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionDocument" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionDocument.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionDocuments" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionDocuments.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getCriticalFunctionPeakPeriods" extends="kcpt.baseCriticalFunctionPage">
        <put-attribute name="criticalFunctionContent" value="/pages/plan/include/getCriticalFunctionPeakPeriods.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getFacultyPreparednessActionItem" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getFacultyPreparednessActionItem.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getFacultyPreparednessActionItems" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getFacultyPreparednessActionItems.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyCentralApplications" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyCentralApplications.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyDepartmentApplication" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyDepartmentApplication.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyDepartmentApplications" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyDepartmentApplications.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyServer" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyServer.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyServers" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyServers.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyWorkstations" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyWorkstations.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyRestart" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyRestart.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyActionItem" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyActionItem.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInformationTechnologyActionItems" extends="kcpt.baseInformationTechnologyPage">
        <put-attribute name="informationTechnologyContent" value="/pages/plan/include/getInformationTechnologyActionItems.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesStaffMembers" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesStaffMembers.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesStaffMember" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesStaffMember.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesWorkFromHomeStaffs" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesWorkFromHomeStaffs.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesWorkFromHomeStaff" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesWorkFromHomeStaff.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesTeams" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesTeams.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesTeamMember" extends="kcpt.basePageNoRightSection">
        <put-attribute name="content" value="/pages/plan/include/getKeyResourcesTeamMember.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesSkills" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesSkills.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesStaffReqs" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesStaffReqs.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesStaffReq" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesStaffReq.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesOtherUnitStaffs" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesOtherUnitStaffs.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesOtherUnitStaff" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesOtherUnitStaff.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesStakeholders" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesStakeholders.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesStakeholder" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesStakeholder.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesDocuments" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesDocuments.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesDocument" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesDocument.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesEquipmentReqs" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesEquipmentReqs.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getKeyResourcesFacilitiesReqs" extends="kcpt.baseKeyResourcesPage">
        <put-attribute name="keyResourcesContent" value="/pages/plan/include/getKeyResourcesFacilitiesReqs.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getPlanActionItem" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getPlanActionItem.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getPlanActionItems" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getPlanActionItems.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructions" extends="kcpt.basePlanPage">
        <put-attribute name="planContent" value="/pages/plan/include/getInstructions.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructionHighPriorityCourses" extends="kcpt.baseInstructionPage">
        <put-attribute name="instructionContent" value="/pages/plan/include/getInstructionHighPriorityCourses.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructionAllCourses" extends="kcpt.baseInstructionPage">
        <put-attribute name="instructionContent" value="/pages/plan/include/getInstructionAllCourses.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructionDepartmentalPractices" extends="kcpt.baseInstructionPage">
        <put-attribute name="instructionContent" value="/pages/plan/include/getInstructionDepartmentalPractices.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructionTeachingIssues" extends="kcpt.baseInstructionPage">
        <put-attribute name="instructionContent" value="/pages/plan/include/getInstructionTeachingIssues.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructionActionItems" extends="kcpt.baseInstructionPage">
        <put-attribute name="instructionContent" value="/pages/plan/include/getInstructionActionItems.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.getInstructionActionItem" extends="kcpt.baseInstructionPage">
        <put-attribute name="instructionContent" value="/pages/plan/include/getInstructionActionItem.jsp" cascade="true"/>
    </definition>

    <definition name="kcpt.adminInstructionScreens" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/admin/include/adminInstructionScreens.jsp"/>
    </definition>

    <definition name="kcpt.adminCriticalityLevels" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/admin/include/adminCriticalityLevels.jsp"/>
    </definition>

    <definition name="kcpt.adminSamplePlan" extends="kcpt.basePage">
        <put-attribute name="content" value="/pages/admin/include/adminSamplePlan.jsp"/>
    </definition>

    <definition name="kcpt.getCritialFunctionGuidanceStepTwo" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/plan/include/getCriticalFunctionGuidanceStepTwo.jsp"/>
    </definition>

    <definition name="kcpt.logout" extends="kcpt.baseLoginPage">
        <put-attribute name="content" value="/pages/login/include/logout.jsp"/>
    </definition>

    <definition name="kcpt.acknowledgments" extends="kcpt.basePageNoRightSection">
        <put-attribute name="content" value="/pages/plan/include/acknowledgments.jsp"/>
    </definition>

</tiles-definitions>

redpajama_set_name: RedPajamaGithub


text:
#include "config.h"
#include "modules/accessibility/AXMenuListPopup.h"

#include "core/html/HTMLSelectElement.h"
#include "modules/accessibility/AXMenuListOption.h"
#include "modules/accessibility/AXObjectCacheImpl.h"

namespace blink {

using namespace HTMLNames;

AXMenuListPopup::AXMenuListPopup(AXObjectCacheImpl& axObjectCache)
    : AXMockObject(axObjectCache), m_activeIndex(-1)
{
}

bool AXMenuListPopup::isVisible() const
{
    return !isOffScreen();
}

bool AXMenuListPopup::isOffScreen() const
{
    if (!m_parent)
        return true;

    return m_parent->isCollapsed();
}

bool AXMenuListPopup::isEnabled() const
{
    if (!m_parent)
        return false;

    return m_parent->isEnabled();
}

bool AXMenuListPopup::computeAccessibilityIsIgnored(IgnoredReasons* ignoredReasons) const
{
    return accessibilityIsIgnoredByDefault(ignoredReasons);
}

AXMenuListOption* AXMenuListPopup::menuListOptionAXObject(HTMLElement* element) const
{
    ASSERT(element);
    if (!isHTMLOptionElement(*element))
        return 0;

    AXObject* object = axObjectCache().getOrCreate(element);
    if (!object || !object->isMenuListOption())
        return 0;

    return toAXMenuListOption(object);
}

int AXMenuListPopup::getSelectedIndex() const
{
    if (!m_parent)
        return -1;

    Node* parentNode = m_parent->node();
    if (!isHTMLSelectElement(parentNode))
        return -1;

    HTMLSelectElement* htmlSelectElement = toHTMLSelectElement(parentNode);
    return htmlSelectElement->selectedIndex();
}

bool AXMenuListPopup::press() const
{
    if (!m_parent)
        return false;

    m_parent->press();
    return true;
}

void AXMenuListPopup::addChildren()
{
    ASSERT(!isDetached());
    if (!m_parent)
        return;

    Node* parentNode = m_parent->node();
    if (!isHTMLSelectElement(parentNode))
        return;

    HTMLSelectElement* htmlSelectElement = toHTMLSelectElement(parentNode);
    m_haveChildren = true;

    if (m_activeIndex == -1)
        m_activeIndex = getSelectedIndex();

    const WillBeHeapVector<RawPtrWillBeMember<HTMLElement>>& listItems = htmlSelectElement->listItems();
    unsigned length = listItems.size();
    for (unsigned i = 0; i < length; i++) {
        AXMenuListOption* option = menuListOptionAXObject(listItems[i]);
        if (option) {
            option->setParent(this);
            m_children.append(option);
        }
    }
}

void AXMenuListPopup::updateChildrenIfNecessary()
{
    if (m_haveChildren && m_parent && m_parent->needsToUpdateChildren())
        clearChildren();

    if (!m_haveChildren)
        addChildren();
}

void AXMenuListPopup::didUpdateActiveOption(int optionIndex)
{
    updateChildrenIfNecessary();

    AXObjectCacheImpl& cache = axObjectCache();
    if (m_activeIndex != optionIndex && m_activeIndex >= 0 && m_activeIndex < static_cast<int>(m_children.size())) {
        AXObject* previousChild = m_children[m_activeIndex].get();
        cache.postNotification(previousChild, AXObjectCacheImpl::AXMenuListItemUnselected);
    }

    if (optionIndex >= 0 && optionIndex < static_cast<int>(m_children.size())) {
        AXObject* child = m_children[optionIndex].get();
        cache.postNotification(child, AXObjectCacheImpl::AXFocusedUIElementChanged);
        cache.postNotification(child, AXObjectCacheImpl::AXMenuListItemSelected);
    }

    m_activeIndex = optionIndex;
}

void AXMenuListPopup::didHide()
{
    AXObjectCacheImpl& cache = axObjectCache();
    cache.postNotification(this, AXObjectCacheImpl::AXHide);
    if (activeChild())
        cache.postNotification(activeChild(), AXObjectCacheImpl::AXMenuListItemUnselected);
}

void AXMenuListPopup::didShow()
{
    if (!m_haveChildren)
        addChildren();

    AXObjectCacheImpl& cache = axObjectCache();
    cache.postNotification(this, AXObjectCacheImpl::AXShow);
    int selectedIndex = getSelectedIndex();
    if (selectedIndex >= 0 && selectedIndex < static_cast<int>(m_children.size()))
        didUpdateActiveOption(selectedIndex);
    else
        cache.postNotification(m_parent, AXObjectCacheImpl::AXFocusedUIElementChanged);
}

AXObject* AXMenuListPopup::activeChild()
{
    if (m_activeIndex < 0 || m_activeIndex >= static_cast<int>(children().size()))
        return nullptr;

    return m_children[m_activeIndex].get();
}

} // namespace blink