In Reinforcement Learning (RL), an agent explores the environment and collects trajectories into the memory buffer for later learning. However, the collected trajectories can easily be imbalanced with respect to the achieved goal states. The problem of learning from imbalanced data is a well-known problem in supervised learning, but has not yet been thoroughly researched in RL. To address this problem, we propose a novel Curiosity-Driven Prioritization (CDP) framework to encourage the agent to over-sample those trajectories that have rare achieved goal states. The CDP framework mimics the human learning process and focuses more on relatively uncommon events. We evaluate our methods using the robotic environment provided by OpenAI Gym. The environment contains six robot manipulation tasks. In our experiments, we combined CDP with Deep Deterministic Policy Gradient (DDPG) with or without Hindsight Experience Replay (HER). The experimental results show that CDP improves both performance and sampleefficiency of reinforcement learning agents, compared to state-of-the-art methods.
This paper addresses a problem that arises in "universal" value-function approximation (that is, reinforcement-learning when a current goal is included as part of the input);  when doing experience replay, the experience buffer might have much more representation of some goals than others, and it's important to keep the training appropriately balanced over goals.
So, the idea is to a kind of importance weighting of the trajectory memory, by doing a density estimation on the goal distribution represented in the memory and then sample them for training in a way that is inversely related to their densities.  This method results in a moderate improvement in the effectiveness of DDPG, compared to the previous method for hindsight experience replay.
The idea is intuitively sensible, but I believe this paper falls short of being ready for publication for three major reasons.
First, the mechanism provided has no mathematical justification--it seems fairly arbitrary.   Even if it's not possible to prove something about this strategy, it would be useful to just state a desirable property that the sampling mechanism should have and then argue informally that this mechanism has that property.  As it is, it's just one point in a large space of possible mechanisms.
I have a substantial concern that this method might end up assigning a high likelihood of resampling trajectories where something unusual happened, not because of the agent's actions, but because of the world having made a very unusual stochastic transition.   If that's true, then this distribution would be very bad for training a value function, which is supposed to involve an expectation over "nature"'s choices in the MDP.
Second, the experiments are (as I understand it, but I may be wrong) in deterministic domains, which definitely does not constitute a general test of a proposed RL  method.  
- I'm not sure we can conclude much from the results on fetchSlide (and it would make sense not to use the last set of parameters but the best one encountered during training)
- What implementation of the other algorithms did you use?
Third, the writing in the paper has some significant lapses in clarity.  I was a substantial way through the paper before understanding exactly what the set-up was;  in particular, exactly what "state" meant was not clear.  I would suggest saying something like s = ((x^g, x^c), g) where s is a state from the perspective of value iteration, (x^g, x^c) is a state of the system, which is a vector of values divided into two sub-vectors, x^g is the part of the system state that involves the state variables that are specified in the goal, x^c (for 'context') is the rest of the system state, and g is the goal.  The dimensions of x^g and g should line up.
- This sentence  was particularly troublesome:  "Each  state s_t also includes the state of the achieved goal, meaning the goal state is a subset of the normal state.  Here, we overwrite the notation s_t  as the achieved goal state, i.e., the state of the object."
- Also, it's important to say what the goal actually is, since it doesn't make sense for it to be a point in a continuous space.  (You do say this later, but it would be helpful to the reader to say it earlier.)