DBS is a therapy for movement disorders such as Parkinson Disease. DBS uses one or two surgically implanted medical devices called neurostimulators, similar to cardiac pacemakers, to deliver electrical stimulation to precisely targeted areas of the brain. One of two areas may be stimulated: either the subthalamic nucleus (STN) or the internal globus pallidus (GPi). These structures are deep within the brain and involved in motor control. A neurologist and a neuro-surgeon decide whether to target the STN or GPi. Stimulation of these areas appears to block the signals that cause the disabling motor symptoms of the disease. As a result, after DBS, many patients achieve greater control over their body movements. The entire system is implanted completely inside the body. Either one or two neurostimulators will be implanted to control symptoms which may affect both sides of the body. DBS leads offering a plurality of electrodes at different depths are commercially available, and allow for the delivery of stimulation using either one electrode or a combination of electrodes.
Presently, DBS therapy is primarily focused on the treatment of Parkinson's Disease, however DBS therapy is expected to expand to the treatment of migraine headaches, schizophrenia, depression, mania and other neurological disorders.
U.S. Pat. No. 6,484,059 issued Nov. 19, 2002 to Gielen et al, the entire contents of which is incorporated herein by reference, teaches an apparatus and method for optimal positioning of a deep brain stimulation electrode for treating movement disorders such as Parkinson's Disease (PD). The method uses two electrodes implanted in two different brain locations, preferably a DBS target such as the Globus Pallidum Internae (GPi) and a feedback target such as the motor cortex (MC). By stimulating both the DBS target and feedback target and observing the relevant patient body movement due to the motor cortex stimulation, the optimal DBS electrode location may be found.
However, after an optimal positioning of the DBS electrode is achieved, optimal stimulation parameters such as stimulation frequency, amplitudes, pulse width or patterns must be determined. Additionally, in the event that a DBS lead with a plurality of electrodes is provided, the appropriate electrode or combination of electrodes must be determined. Unfortunately, optimal stimulation patterns vary from patient to patient and generally are programmable parameters of an implantable pulse generator (IPG). Hence, in the prior art a clinician needs to adjust the stimulation parameters after implantation and perform regular follow up procedures. Such follow up procedures are both costly and inconvenient.
U.S. Pat. No. 7,006,872 issued Feb. 28, 2006 to Gielen et al, the entire contents of which is incorporated herein by reference, teaches a system and method for predicting the likelihood of occurrence of an impending neurological episode. Electrical stimuli are delivered to a structure of the brain. Response field potentials evoked by the stimuli are sensed. Analysis of these field potentials allows for predictions as to the occurrence of an impending, but not yet occurring, neurological disorder. In one example, a measurement of change in response pulses is used to determine a level of interconnectivity in the structures of the brain. The level of functional interconnectivity is used in predicting the occurrence of the neurological event. An example of such a neurological event includes an epileptic seizure. In summary, from the changes in the response of the brain activity to the two stimulations a decision can be made regarding an impending epilepsy episode and an appropriate stimulation therapy can be delivered.
Gielen, in the above mentioned U.S. patent, further teaches that the threshold value defining a likely neurological episode may be a self learning process. The pulse therapy is defined by a clinician at time of implantation and is updated as needed. The sensed signals may be stored on a memory for retrieval by the clinician for therapy assessment. Treatment is set by a clinician, and only the triggering mechanism is learned.
U.S. Pat. No. 7,231,254 issued Jun. 12, 2007 to Dilorenzo, the entire contents of which is incorporated herein by reference, teaches a closed loop system for neuromodulation in general and more specifically also a closed loop deep brain stimulation control system for Parkinson patients. The closed loop controllers of Dilorenzo deliver neuromodulation responsive to treatment parameters derived from a neural response to previously delivered neural modulation signals sensed by one or more sensors. However Dilorenzo does not describe using a machine learning scheme to find optimal stimulation parameters derived such that the control system will achieve the optimal stimulation therapy on line and without prior knowledge of patient characteristics and in different patient daily life activities.
Thus, according to the prior art the complicated task of finding the optimal lead position, and/or combination of leads, in implantation and optimizing the IPG stimulation parameters are done manually by the clinician during implantation and patient follow up and with no systematic method to ensure that optimal therapy will be delivered.
Q-learning (QL) is a reinforcement learning technique that works by learning an action-value function that gives the expected utility of taking a given action in a given state and following a fixed policy thereafter. One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment. Watkins and Dayan in an article entitled “Q Learning”, published 1992 in Machine Learning 8, 279-292, 1992, showed that online solution of a QL recursive formula is guaranteed to converge to the optimal policy in a model free reinforcement learning problem.
There is therefore a long felt need to develop a systematic closed loop DBS device control system that will achieve optimal stimulation therapy and wherein parameters for appropriate stimulation therapy are controlled automatically by a machine learning, module without requiring constant manual patient follow up.