Some real-world domains are best characterized as a single task, but for others this perspective is limiting. Instead, some tasks continually grow in complexity, in tandem with the agent's competence. In continual learning there are no explicit task boundaries or curricula. As learning agents have become more powerful, continual learning remains one of the frontiers that has resisted quick progress. To test continual learning capabilities we consider a challenging 3D domain with an implicit sequence of tasks and sparse rewards. We propose a novel agent architecture called Unicorn, which demonstrates strong continual learning and outperforms several baseline agents on the proposed domain. The agent achieves this by jointly representing and efficiently learning multiple policies for multiple goals, using a parallel off-policy learning setup.
Summary
The authors aim to do continual learning to solve dependent tasks using "single-stage end-to-end learning". The resulting "Unicorn" agent trains on all tasks simultaneously. The idea is to use multi-task "off-policy learning", which uses (old) trajectories (experience) from task A to help learning on a related task B. Authors further distinguish between goals (inputs to Q) and tasks (different reward functions). A goal might a color/shape of an object to pick up.
The core model is a UVFA that learns a goal-conditioned Q-function Q(s,a,g). 
Some technical aspects: 
- use n-step returns.
- when training Q on goal g_i, authors also trajectories that were generated using Q conditioned on another goal(s) g_j. They then truncate the returns for task i when an action taken is not optimal under Q(s,a,g_j) conditioned on goal j. The intuition (seems) to be that this 
- authors do not use experience replay or a target Q-function, since the parallelized implementation is reported to be stable enough.
- unicorn sees all train task reward functions during training (but not hold out task rewards).
- unicorn is tested on several 3d maze environments with key-lock etc semantics. The tasks / goals seem simple, and the dependency is defined by changing colors / shapes of objects to be picked up. Authors argue unicorn has to learn to relate task rewards to these goal features.
- unicorn is compared against baselines that 1) do single-task learning (expert) 2) learn on a sum of task rewards (glutton), 3) uniformly random baseline. 
- authors show that 1) unicorn performs better on train tasks 2) performs better on hold-out tasks. Also, authors show results for zero-shot transfer learning, with adding abstract tasks (extra reward for picking up any object) improving performance, 
Pro
- Simple approach (e.g., no experience replay etc), and uses only a limited set of techniques (e.g., reward truncation). 
- Reward performance suggests the model has more properly related goal features to different payoffs.
- Analysis of qualitative behavior is nice.
Con
- The writing is a bit dense in places, e.g., the discussion of baselines is a bit hard to read.
- Description of algorithm is wrapped in long text, a clear algorithm box would make the approach much clearer.
- Not clear what kind of hyperparameters are introduced / used / tuned for Unicorn. 
- Authors say "deep dependency", but this seems to just refer to different colors / shapes between objects in the env used in the paper. How is "dependency" between goals and tasks defined in general? 
- The experimental setting seems a bit limited, authors only show results on a single domain, and do not offer rigorous definitions. This makes the scope of the paper rather limited.  
Reproducibility: 
- It's not clear what the variance in the baseline performance is (variance only shown for unicorn).