Answers.Body: <p>A first question that I think is important to consider is: <strong>do you expect the data that you're dealing with to be changing over time</strong> (i.e. do you expect there to be <em>concept drift</em>)? This could be any kind of change. Simply changes in how frequent certain inputs are, changes in how frequent positives/negatives are, or even changes in relations between inputs and ground-truth positive/negative labels.</p>
  
  <p>If you do <strong>not</strong> expect there to be concept drift, I'd almost consider suggesting that you may not have that big of a problem. It might be worth not doing anything at all with the data you receive online, and just stick to what you learned initially from offline data. Or you could try to use those few extra predicted-positive samples that you get for finetuning. You'd just have to be careful not too change your model too much based on this, because you know that you're not receiving a representative sample of <strong>all</strong> the data anymore here, so you might bias your model if you pay too much attention to only this online data relative to the offline data.</p>
  
  <hr>
  
  <p>I guess the question becomes much more interesting if you do expect there to be concept drift, and it also seems likely that you are indeed dealing with this in most of the situations that would match the problem description. In this case, you will indeed want to make good use of the new data that you get online, because it can allow you to adapt to changes in the data that you're dealing with.</p>
  
  <p>So, one "solution" could be to just... ignore the problem that you're only learning online from a biased sample of all your data (only from the predicted-positives), and just learn anyway. This might actually not perform too badly. Unless your model is really incredibly good already, you'll likely still get false positives, and so also still be able to learn from some of those -- you're not learning exclusively from positives. Still, the false positives won't be representative of <strong>all</strong> the negatives, so you still have bias. </p>
  
  <p>The only better solution I can think of is relaxing this assumption:</p>
  
  <blockquote>
    <p>Once deployed, humans review and check for correctness <em>only items predicted positive</em> by the algorithm. </p>
  </blockquote>
  
  <p>You can still have the humans focus on predicted positives, but maybe have them inspect a predicted-negative also sometimes. Not often, just a few times. You can think of this as doing <strong>exploration</strong> like you would in reinforcement learning settings. You could do it randomly (randomly pick predicted negatives with some small probability), but you could also be smarter about it and explicitly target exploration of instances that your model is "unsure" about, or instances that are unlike data you've seen before (to specifically target concept drift).</p>
  
  <p>I have a paper about something <strong>very similar</strong> to this right here: <a href="https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16183" rel="nofollow noreferrer">Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees</a>. Here the assumption is that we're dealing with (potentially fraudulent) transactions, of which we can pick out and manually inspect a very small sample online. The only real difference in this paper is that we assumed that different transactions also had different monetary "rewards" for getting correctly caught as positives, based on the transaction amount. So a transaction of a very high amount could be worth inspecting even if we predicted a low probability of being fraudulent, whereas a transaction of a very low amount might be ignored even if it had a higher predicted probability of being fraudulent.</p>
  
  <hr>
  
  <blockquote>
    <p>what's a meaningful metric to monitor to ensure that the performance of the model is not degrading?* (given the constraint specified here, F1 score is unknown).</p>
  </blockquote>
  
  <p>Having a labelled evaluation set for this could be useful if possible... but it also might not be representative if <strong>concept drift</strong> is expected to be a major issue in your problem setting (because I suppose that the concept drift that you deal with online would not be reflected in an older, labelled evaluation set).</p>
  
  <p>Just keeping track of things that you can measure online, like precision, and how it changes over time, could be useful enough already. With some additional assumptions, you could get rough estimates of other metrics. For instance, if you assume that the ratio <span class="math-container">$\frac{TP + FN}{FP + TN}$</span> between ground-truth-positives and ground-truth-negatives remains constant (remains the same as it was in your offline, labelled data), you could also try to extrapolate approximately how many positives you've missed out on. If your precision is dropping over time (your true positives are getting lower), you know -- assuming that fraction remains constant -- that your false negatives somewhere else in the dataset must be growing by approximately the same absolute number.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T19:02:46.640Z
Answers.LastActivityDate: 2020-06-03T19:02:46.640Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I have a binary classifier (think of it as a content moderation system) that is deployed after having being trained via batch learning.</p>
  
  <p>Once deployed, humans review and check for correctness <em>only items predicted positive</em> by the algorithm. </p>
  
  <p>In other words, once in production if I group predictions of the model on unseen examples in the confusion matrix </p>
  
  <pre><code>+-----------+-----------------+
  |           |   Ground-truth  |
  |           +-----+-----------+
  |           |     | Neg | Pos |
  +-----------+-----+-----+-----+
  |           | Neg | x11 | x12 |
  | Predicted +-----+-----+-----+
  |           | Pos | x21 | x22 |
  +-----------+-----+-----+-----+
  </code></pre>
  
  <ul>
  <li>I have access to all the ground-truth labels of the elements counted in <span class="math-container">$x_{21}$</span>, <span class="math-container">$x_{22}$</span> (the predicted-positive)</li>
  <li>I know the <em>sum</em> of <span class="math-container">$x_{11}$</span> and <span class="math-container">$x_{12}$</span>, but not their values</li>
  <li>I do <em>not</em> have access to the ground-truth labels of the elements predicted-negative.</li>
  </ul>
  
  <p>This (suboptimal) setup allows to measure precision <span class="math-container">$\frac{x_{22}}{x_{21} + x_{22}}$</span>, while recall stays unknown as elements predicted negative are not examined at all (ground-truth labels of negatives can't be assigned due to resource constraints). </p>
  
  <p>Information gathered from users about the (true and false) positive elements can be used to feed a retraining loop... but</p>
  
  <ol>
  <li>are there any "smart" learning recipes that are expected to make the algorithm improve its overall performance (say, the F1 score for the positive class) in this setting?</li>
  <li>what's a meaningful metric to monitor to ensure that the performance of the model is not <em>degrading</em>?* (given the constraint specified here, F1 score is unknown).</li>
  </ol>
  
  <p>Thanks for any hint on how to deal with this!</p>
  
  <p><sub> * One solution could be to continuously monitor the F1 score on a labeled evaluation set, but maybe there's more one can do?</sub></p>
  
Tags: <machine-learning><online-learning><algorithmic-bias>
Title: How do I keep my system (online) learning if I can get ground truth labels only for examples flagged positive?
UserDisplayName: Davide Fiocco
ThreadText: # How do I keep my system (online) learning if I can get ground truth labels only for examples flagged positive?
  
  (Asked by: username_0 on 2020-06-01)
  
  I have a binary classifier (think of it as a content moderation system) that is deployed after having being trained via batch learning.
  Once deployed, humans review and check for correctness only items predicted positive by the algorithm. 
  In other words, once in production if I group predictions of the model on unseen examples in the confusion matrix 
  +-----------+-----------------+
  |           |   Ground-truth  |
  |           +-----+-----------+
  |           |     | Neg | Pos |
  +-----------+-----+-----+-----+
  |           | Neg | x11 | x12 |
  | Predicted +-----+-----+-----+
  |           | Pos | x21 | x22 |
  +-----------+-----+-----+-----+
  
  
  I have access to all the ground-truth labels of the elements counted in $x_{21}$, $x_{22}$ (the predicted-positive)
  I know the sum of $x_{11}$ and $x_{12}$, but not their values
  I do not have access to the ground-truth labels of the elements predicted-negative.
  
  This (suboptimal) setup allows to measure precision $\frac{x_{22}}{x_{21} + x_{22}}$, while recall stays unknown as elements predicted negative are not examined at all (ground-truth labels of negatives can't be assigned due to resource constraints). 
  Information gathered from users about the (true and false) positive elements can be used to feed a retraining loop... but
  
  are there any "smart" learning recipes that are expected to make the algorithm improve its overall performance (say, the F1 score for the positive class) in this setting?
  what's a meaningful metric to monitor to ensure that the performance of the model is not degrading?* (given the constraint specified here, F1 score is unknown).
  
  Thanks for any hint on how to deal with this!
   * One solution could be to continuously monitor the F1 score on a labeled evaluation set, but maybe there's more one can do?
  
  **Tags:** machine-learning, online-learning, algorithmic-bias
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-03. Score: 3
  
  A first question that I think is important to consider is: **do you expect the data that you're dealing with to be changing over time** (i.e. do you expect there to be *concept drift*)? This could be any kind of change. Simply changes in how frequent certain inputs are, changes in how frequent positives/negatives are, or even changes in relations between inputs and ground-truth positive/negative labels.
  
  If you do **not** expect there to be concept drift, I'd almost consider suggesting that you may not have that big of a problem. It might be worth not doing anything at all with the data you receive online, and just stick to what you learned initially from offline data. Or you could try to use those few extra predicted-positive samples that you get for finetuning. You'd just have to be careful not too change your model too much based on this, because you know that you're not receiving a representative sample of **all** the data anymore here, so you might bias your model if you pay too much attention to only this online data relative to the offline data.
  
  ---
  
  I guess the question becomes much more interesting if you do expect there to be concept drift, and it also seems likely that you are indeed dealing with this in most of the situations that would match the problem description. In this case, you will indeed want to make good use of the new data that you get online, because it can allow you to adapt to changes in the data that you're dealing with.
  
  So, one "solution" could be to just... ignore the problem that you're only learning online from a biased sample of all your data (only from the predicted-positives), and just learn anyway. This might actually not perform too badly. Unless your model is really incredibly good already, you'll likely still get false positives, and so also still be able to learn from some of those -- you're not learning exclusively from positives. Still, the false positives won't be representative of **all** the negatives, so you still have bias.
  
  The only better solution I can think of is relaxing this assumption:
  
  > Once deployed, humans review and check for correctness *only items predicted positive* by the algorithm.
  
  You can still have the humans focus on predicted positives, but maybe have them inspect a predicted-negative also sometimes. Not often, just a few times. You can think of this as doing **exploration** like you would in reinforcement learning settings. You could do it randomly (randomly pick predicted negatives with some small probability), but you could also be smarter about it and explicitly target exploration of instances that your model is "unsure" about, or instances that are unlike data you've seen before (to specifically target concept drift).
  
  I have a paper about something **very similar** to this right here: [Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16183). Here the assumption is that we're dealing with (potentially fraudulent) transactions, of which we can pick out and manually inspect a very small sample online. The only real difference in this paper is that we assumed that different transactions also had different monetary "rewards" for getting correctly caught as positives, based on the transaction amount. So a transaction of a very high amount could be worth inspecting even if we predicted a low probability of being fraudulent, whereas a transaction of a very low amount might be ignored even if it had a higher predicted probability of being fraudulent.
  
  ---
  
  > what's a meaningful metric to monitor to ensure that the performance of the model is not degrading?\* (given the constraint specified here, F1 score is unknown).
  
  Having a labelled evaluation set for this could be useful if possible... but it also might not be representative if **concept drift** is expected to be a major issue in your problem setting (because I suppose that the concept drift that you deal with online would not be reflected in an older, labelled evaluation set).
  
  Just keeping track of things that you can measure online, like precision, and how it changes over time, could be useful enough already. With some additional assumptions, you could get rough estimates of other metrics. For instance, if you assume that the ratio $\frac{TP + FN}{FP + TN}$ between ground-truth-positives and ground-truth-negatives remains constant (remains the same as it was in your offline, labelled data), you could also try to extrapolate approximately how many positives you've missed out on. If your precision is dropping over time (your true positives are getting lower), you know -- assuming that fraction remains constant -- that your false negatives somewhere else in the dataset must be growing by approximately the same absolute number.
  
  
  ---
  
  