Answers.Body: <p>It is quite often <em>possible</em> to frame a problem as a Reinforcement Learning (RL) problem at some level. However, this may turn out to be for no benefit, or a net cost towards solving the problem. Casting parameter or hyperparameter searches as RL can be adding a layer of complexity and reduce efficiency.</p>
  
  <p>One key thing to bear in mind is that any classification or regression that occurs within a RL framework will end up using effectively the same models and approaches that could solve the same problems <em>directly</em>. These models would either appear directly as the function approximators in RL that implement policies or value functions, or they would be an implied part of them. If you have labeled data for classification - even delayed until some time after you collect data, then you are usually going to be better off using supervised learning directly. </p>
  
  <p>For hyperparameter searches (e.g. cutoffs for anomaly detection) then you may not need labelled data, but just need a good way to test the model offline.</p>
  
  <p>The first point at which supervised learning or classic anomaly detection might fail for you is if you never receive any feedback about individual records, only a measure of overall performance. In other words, if you can measure <em>consequences</em> of good performance, but never measure or check  <em>correctness</em>.</p>
  
  <p>Your characterisation</p>
  
  <blockquote>
    <p>about 7-10% of data do not make any sense.</p>
  </blockquote>
  
  <p>does not appear to fit that. It looks like you could detect this, maybe manually labelling a few thousand records, and train a classifier using supervised learning techniques. That is likely a much better use of your time than trying to restructure the problem at a higher level and trusting a trail-and-error approach to discover the same rules.</p>
  
  <p>Putting that to one side, assuming you do have a problem where</p>
  
  <ul>
  <li>data to be classified is arriving as a stream, and needs to be processed online, item by item or in small batches</li>
  <li>you have reason to think that an accept/reject stage before processing further would be useful</li>
  <li>you have no way to label training data for accept/reject</li>
  <li>you have a way to measure performance of the remaining system after the accept/reject phase</li>
  </ul>
  
  <p>then you could use RL to frame the accept/reject phase as an action. There are some challenges there, but essentially you would use RL along with measurement feedback to sample errors or gradients - typically using TD Error or policy gradients. This could wrap almost any model that does classification or anomaly detection etc, provided it could be trained using those gradients.</p>
  
  <p>From comments, if the underlying distributions for accept/reject are non-stationary, this may point you more towards a RL solution. However, that may come with a cost to performance - you will need to balance exploration rate (which will reduce the performance of the model against stationary data) versus speed of learning new distributions. This is a problem for all online learners; the main advantage of a RL approach here is that it will not require generating new labelled data. If you can use a recency-weighted anomaly detection algorithm instead, then you won't need the labeled data either - whether that is better requires testing, personally I'd take a working anomaly detection as the baseline and only use RL if it proved itself better.</p>
  
  <p>The specific items that you turn into states and rewards are not clear to me from the question, and you would need to work on these things carefully. It is possible you will need more than the current data item in order to define state, and that will depend a lot on how the feedback loop works that establishes reward. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-01T14:51:38.940Z
Answers.Text: Regarding classification and labelling, there still might be a problem with data which have a trend or which might have non-stationary distribution. RL approach would help to adapt to changes in the underlying data.
Answers.UserDisplayName: Artem Oboturov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-01T15:13:15.650Z
Answers.Text: @ArtemOboturov: It *might*, but at the expense of needing to increase exploration, which would impact performance of the system as the agent would need to make more deliberate misclassifications over time in order to test whether distributions had changed. Most RL agents do at least have mechanisms for that - e.g. some form of Upper Confidence Bound - whether it is worth doing depends on the degree of drift you expect, and how happy you would be to allow this routine testing to affect your data stream.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-01T15:18:56.030Z
Answers.Text: Also, if you find that you can rely on unlabelled anomaly detection, then provided there is always a reasonable distance between acceptable and reject-able data, you can probably just use a simple online version of that - e.g. base the acceptance scores for mean and standard deviation on recency-weighted statistics.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-01T15:26:31.887Z
Answers.Text: @ArtemOboturov I have added a paragraph about non-stationary problems to the answer.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-01T12:03:31.503Z
Answers.LastActivityDate: 2018-10-01T15:25:42.310Z
Answers.LastEditDate: 2018-10-01T15:25:42.310Z
Answers.UserDisplayName: Neil Slater
Body: <p>To me it seems to be ill defined. Partially because of absence of knowledge which points are to be considered outliers in the first place.</p>
  
  <p>The problem which I have in mind is "bad market data" detection. For example if a financial data provider is good only most of the time, but about 7-10% of data do not make any sense.</p>
  
  <p>The action space is binary: either take an observation or reject it.</p>
  
  <p>I am not sure about the reward, because the observations would be fed into an algorithm as inputs and the outputs of the algo would be outliers themselves. So the outliers detection should prevent outputs of the algorithm going rouge.</p>
  
  <p>It is necessary to add that if we are talking about the market data (stocks, indices, fx), there's no guarantee that the distributions are stationary and there might be trends and jumps. If a supervised classifier is trainer based on historical data, how and how often should it be adjusted to be able to cope with different modes of the data.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-01T08:33:34.573Z
Comments.Text: Reinforcement learning is not a category of problems. It is a solution method for Markov decision processes (MDPs). The question should be *"Can outlier detection be formulated as an MDP?"*.
Comments.UserDisplayName: Hai Nguyen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-01T08:39:09.570Z
Comments.Text: Thank you for the comments. The description was expanded to address question from @NeilSlater.
Comments.UserDisplayName: Artem Oboturov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-01T10:12:42.857Z
Comments.Text: @HaiNguyen: The additional factor that makes something a "RL problem" as opposed to an MDP formulation is the learning/optimisation goal, such as the definition of return, discount factor, whether to use average reward, whether we aim for optimal control or measuring a fixed policy. Otherwise agreed, a large part of the OP's problem would be about formulating their scenario as a MDP (or more likely *framing* it as one at a higher level due to consequences of choices). However, I don't see a need to re-word that part of the question
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning>
Title: Is it possible to state an outliers detection problem as a reinforcement learning problem?
UserDisplayName: Artem Oboturov
ThreadText: # Is it possible to state an outliers detection problem as a reinforcement learning problem?
  
  (Asked by: username_0 on 2018-09-30)
  
  To me it seems to be ill defined. Partially because of absence of knowledge which points are to be considered outliers in the first place.
  
  The problem which I have in mind is "bad market data" detection. For example if a financial data provider is good only most of the time, but about 7-10% of data do not make any sense.
  
  The action space is binary: either take an observation or reject it.
  
  I am not sure about the reward, because the observations would be fed into an algorithm as inputs and the outputs of the algo would be outliers themselves. So the outliers detection should prevent outputs of the algorithm going rouge.
  
  It is necessary to add that if we are talking about the market data (stocks, indices, fx), there's no guarantee that the distributions are stationary and there might be trends and jumps. If a supervised classifier is trainer based on historical data, how and how often should it be adjusted to be able to cope with different modes of the data.
  
  **Tags:** reinforcement-learning
  
  > **Comments:**
  > **username_1**: Reinforcement learning is not a category of problems. It is a solution method for Markov decision processes (MDPs). The question should be *"Can outlier detection be formulated as an MDP?"*.  
  > **username_0**: Thank you for the comments. The description was expanded to address question from @username_2.  
  > **username_2**: @username_1: The additional factor that makes something a "RL problem" as opposed to an MDP formulation is the learning/optimisation goal, such as the definition of return, discount factor, whether to use average reward, whether we aim for optimal control or measuring a fixed policy. Otherwise agreed, a large part of the OP's problem would be about formulating their scenario as a MDP (or more likely *framing* it as one at a higher level due to consequences of choices). However, I don't see a need to re-word that part of the question  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-10-01. Score: 1
  
  It is quite often *possible* to frame a problem as a Reinforcement Learning (RL) problem at some level. However, this may turn out to be for no benefit, or a net cost towards solving the problem. Casting parameter or hyperparameter searches as RL can be adding a layer of complexity and reduce efficiency.
  
  One key thing to bear in mind is that any classification or regression that occurs within a RL framework will end up using effectively the same models and approaches that could solve the same problems *directly*. These models would either appear directly as the function approximators in RL that implement policies or value functions, or they would be an implied part of them. If you have labeled data for classification - even delayed until some time after you collect data, then you are usually going to be better off using supervised learning directly.
  
  For hyperparameter searches (e.g. cutoffs for anomaly detection) then you may not need labelled data, but just need a good way to test the model offline.
  
  The first point at which supervised learning or classic anomaly detection might fail for you is if you never receive any feedback about individual records, only a measure of overall performance. In other words, if you can measure *consequences* of good performance, but never measure or check *correctness*.
  
  Your characterisation
  
  > about 7-10% of data do not make any sense.
  
  does not appear to fit that. It looks like you could detect this, maybe manually labelling a few thousand records, and train a classifier using supervised learning techniques. That is likely a much better use of your time than trying to restructure the problem at a higher level and trusting a trail-and-error approach to discover the same rules.
  
  Putting that to one side, assuming you do have a problem where
  
  * data to be classified is arriving as a stream, and needs to be processed online, item by item or in small batches
  * you have reason to think that an accept/reject stage before processing further would be useful
  * you have no way to label training data for accept/reject
  * you have a way to measure performance of the remaining system after the accept/reject phase
  
  then you could use RL to frame the accept/reject phase as an action. There are some challenges there, but essentially you would use RL along with measurement feedback to sample errors or gradients - typically using TD Error or policy gradients. This could wrap almost any model that does classification or anomaly detection etc, provided it could be trained using those gradients.
  
  From comments, if the underlying distributions for accept/reject are non-stationary, this may point you more towards a RL solution. However, that may come with a cost to performance - you will need to balance exploration rate (which will reduce the performance of the model against stationary data) versus speed of learning new distributions. This is a problem for all online learners; the main advantage of a RL approach here is that it will not require generating new labelled data. If you can use a recency-weighted anomaly detection algorithm instead, then you won't need the labeled data either - whether that is better requires testing, personally I'd take a working anomaly detection as the baseline and only use RL if it proved itself better.
  
  The specific items that you turn into states and rewards are not clear to me from the question, and you would need to work on these things carefully. It is possible you will need more than the current data item in order to define state, and that will depend a lot on how the feedback loop works that establishes reward.
  
  > **Comments:**
  > **username_0**: Regarding classification and labelling, there still might be a problem with data which have a trend or which might have non-stationary distribution. RL approach would help to adapt to changes in the underlying data.  
  > **username_2**: @username_0: It *might*, but at the expense of needing to increase exploration, which would impact performance of the system as the agent would need to make more deliberate misclassifications over time in order to test whether distributions had changed. Most RL agents do at least have mechanisms for that - e.g. some form of Upper Confidence Bound - whether it is worth doing depends on the degree of drift you expect, and how happy you would be to allow this routine testing to affect your data stream.  
  > **username_2**: Also, if you find that you can rely on unlabelled anomaly detection, then provided there is always a reasonable distance between acceptable and reject-able data, you can probably just use a simple online version of that - e.g. base the acceptance scores for mean and standard deviation on recency-weighted statistics.  
  > **username_2**: @username_0 I have added a paragraph about non-stationary problems to the answer.  
  
  ---
  
  