Body: <p>I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.</p>
  <p>The details are as follows.</p>
  <ul>
  <li>Each time step is a random batch from the dirty dataset.</li>
  <li>Actions: the cleaning methods (KNN imputer, median imputer etc.)</li>
  <li>loss = sum(-log(selected cleaners' probabilities)) * reward</li>
  <li>reward = predictor score - baseline score</li>
  <li>Predictor score: The score obtained on a clean validation set, after the target task model trained on the cleaned batch with chosen cleaners by the PC network.</li>
  <li>Baseline score: The validation score obtained by training the target task model on ground truth clean dataset.</li>
  </ul>
  <p>After the training is completed, the policy gradient network does not always converge to the best cleaner / combination of cleaners.</p>
  <p>I know that policy gradients are problematic when there are negative rewards, and in my case the reward is mostly negative, as the baseline score is obtained on the ground truth clean dataset.
  I'm also feeding multiple feature vectors of erroneous rows as state vectors to the PG neural network, which could lead to ignoring the appropriate cleaners for single samples -or even penalize them- when a low reward is obtained for the current batch.</p>
  <p>In <a href="https://arxiv.org/abs/1909.11671" rel="nofollow noreferrer">DVRL</a>, they also use batches to evaluate the samples at each iteration (all at once). I believe, in such cases, the minimum loss may not mean the maximum reward. But what could be done to improve the final reward?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-22T19:32:11.857Z
Comments.Text: Could you clarify what's are the actions? Is the network outputting one cleaning method for the whole batch, or potentially a different cleaning method per input in the dataset?
Comments.UserDisplayName: Raphael Lopez Kaufman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-23T11:22:05.403Z
Comments.Text: @RaphaelLopezKaufman Sir, the network outputs one cleaner per input in a batch. I didn't tell about the input states, but I've used the feature vectors generated by error detection methods or some custom features that I created related to metadata.
Comments.UserDisplayName: aby
Tags: <reinforcement-learning><policy-gradients><rewards>
Title: How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
UserDisplayName: aby
ThreadText: # How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
  
  (Asked by: username_0 on 2022-09-21)
  
  I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
  
  The details are as follows.
  
  * Each time step is a random batch from the dirty dataset.
  * Actions: the cleaning methods (KNN imputer, median imputer etc.)
  * loss = sum(-log(selected cleaners' probabilities)) \* reward
  * reward = predictor score - baseline score
  * Predictor score: The score obtained on a clean validation set, after the target task model trained on the cleaned batch with chosen cleaners by the PC network.
  * Baseline score: The validation score obtained by training the target task model on ground truth clean dataset.
  
  After the training is completed, the policy gradient network does not always converge to the best cleaner / combination of cleaners.
  
  I know that policy gradients are problematic when there are negative rewards, and in my case the reward is mostly negative, as the baseline score is obtained on the ground truth clean dataset.
  I'm also feeding multiple feature vectors of erroneous rows as state vectors to the PG neural network, which could lead to ignoring the appropriate cleaners for single samples -or even penalize them- when a low reward is obtained for the current batch.
  
  In [DVRL](https://arxiv.org/abs/1909.11671), they also use batches to evaluate the samples at each iteration (all at once). I believe, in such cases, the minimum loss may not mean the maximum reward. But what could be done to improve the final reward?
  
  **Tags:** reinforcement-learning, policy-gradients, rewards
  > **Comments:**
  > **username_1**: Could you clarify what's are the actions? Is the network outputting one cleaning method for the whole batch, or potentially a different cleaning method per input in the dataset?  
  > **username_0**: @RaphaelLopezKaufman Sir, the network outputs one cleaner per input in a batch. I didn't tell about the input states, but I've used the feature vectors generated by error detection methods or some custom features that I created related to metadata.  
  
  ---
  
  ## 0 answers
  
  