How can deep learning systems flexibly reuse their knowledge? Toward this goal, we propose a new class of challenges, and a class of architectures that can solve them. The challenges are meta-mappings, which involve systematically transforming task behaviors to adapt to new tasks zero-shot. The key to achieving these challenges is representing the task being performed in such a way that this task representation is itself transformable. We therefore draw inspiration from functional programming and recent work in meta-learning to propose a class of Homoiconic Meta-Mapping (HoMM) approaches that represent data points and tasks in a shared latent space, and learn to infer transformations of that space. HoMM approaches can be applied to any type of machine learning task, including supervised learning and reinforcement learning. We demonstrate the utility of this perspective by exhibiting zero-shot remapping of behavior to adapt to new tasks.
This paper introduces a zero-shot task adaptation methodology by learning a meta-learner that reuses a latent representation of data points and tasks (Homoiconic Meta-Mapping). The proposed approach is applied to learn multivariate polynomial, a card game and the paper claims that such a method can apply in both supervised and RL settings.
+ves:
+ Task-to-task mapping is a relevant topic in the contemporary context, and the paper seeks to address this relevant problem.
+ The overall idea of considering winning-losing strategies in meta-learning is interesting.
+ The experimental settings of card games and polynomials are interesting to study the proposed setting, and the results are promising.
Concerns:
- A key issue is that the paper claims to be the first to propose a method for zero-shot task adaptation (pg 8, para before “Future Directions”). The paper seems to have missed papers such as (1) Zero-Shot Task Transfer, Pal et. al, CVPR 2019, and, (2) Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning, Junhyuk Oh et al, ICML 2017. In particular, the CVPR 2019 paper proposes zero-shot task transfer using meta-mappings. Citing these papers and comparing against them is essential, considering this paper addresses the same objective.
- The proposed methodology does not clearly detail how the zero-shot problem setting is used. Section F.2 (Appendix) gives the formulation of a meta-learner and zero-shot task a bit. However, how the meta-learner performs zero-shot task parameter regression is not clear. The notations used in mathematical equations are also not very clear, and a reader needs to swap between different sections to understand the proposed method. (Moving Sec F.2 to the main paper would be very useful. Similarly, moving Figs 5, 7, 8 etc. to the main paper would have helped greatly too.)
- How is the notion of a “task” defined? Are different classes of a dataset tasks (already done in [3]), or are different vision-related objectives as tasks (reference [1])? Are language-related objectives (such as: document classification, sentiment analysis, Named-entity recognition etc.) tasks? In an RL setting, do zero-shot tasks deal with a new environment? It would have been nice to see clearer definitions for the kind of tasks that this method is suited to.
- It was not clear on how the relationship between source tasks and zero-shot tasks are modeled or computed. Clearly, the performance of such a system depends on the correlations between the tasks, and these details are not clear, especially in generalizing the proposed method. The paper seems to correlate tasks and language-related hints; however getting a language hint is not always available, especially for datasets such as Taskonomy [1, see Refs below] - which is a dataset for multiple tasks.
- The zero-shot part is described using a winning-losing scenario, where we have a partial/side knowledge of winning of task_1 and the model can predict losing strategy. Now, what if:
We do not have any knowledge of the winning strategy (or losing strategy) of a zero-shot task?
Can we predict the losing strategy of a game, let’s say card game, from the knowledge of other games like: soccer, tennis, badminton etc. (the initial motivation of the paper). This is quite possible in the continual learning scenario (the paper mentioned some studies on continual learning). If so, then please comment on the negative transfers in that scenario and how it will affect the inference of the meta-learner.
The proposed method also mentioned language descriptions. In case of tasks that are not easy to describe (say, a vision task such as segmentation), how will the proposed method will compute the task-prior of seen-tasks and zero-shot tasks?
The paper mentions “winning-losing” strategy and the metanetwork gives losing strategy parameters. What if the scenario is the otehr way - given a losing strategy, we now have to perform the winning strategy (IMHO this has more value in the real world). Does the paper show the result on this, or am I missing something?
- Experiments are limited to synthetic data and card games, and it would be nice to see the validation of the proposed method on different domains. The notion of zero-shot tasks are also not clear for multivariate polynomial experiment and card-game experiment.
- Figure 1: please clearly mention in the caption what is the meaning and purpose of probes. It is not completely clear whether probes (also referred as held-out set In figure 1 caption) are taking part in training, or only at the time of validation?
- Shared Z vs. Separate task embedding: reuse of M (or, H network) network will only be possible if the input and output dimensions are same - i.e. the dimension of Z_output of M is the same as the dimension of Z_input of the M. What if  input and output dimensions are not the same?
Other minor comments:
* One general comment is: please mention the “Zero-shot” experiments more clearly (may be a separate paragraph or subsection), otherwise it is hard to follow.
* Figure 2(a) please remove the question mark (?) after Held-out. Similarly, Figure 5: Please complete the sentence “ Performing from …”. 
* Figure 1: (a)  Basic meta-learning: Please use some other color to highlight “(a)  Basic meta-learning”, the grey gives an illusion that the step ((a)  Basic meta-learning is not important.
* A term such as “Homoiconic” would have benefited from a formal definition, especially considering it’s not a commonly used term.
* The presentation is in general not well-polished, and needs revision.
References:
[1] Taskonomy, CVPR 2018, Zamir et al
[2] Zero-shot Task Transfer, CVPR 2019, Pal et al
[3] Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer, CVPR 2009, Lampert et al.
[4] Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning, ICML 2017, Junhyuk Oh et al.
======= POST-REBUTTAL COMMENTS=========
I thank the authors for the response. Some of the queries/concerns in the review comments above were clarified. I am updating my rating to Weak Reject, but unfortunately am not completely convinced about accepting the paper in its current form for the following reasons:
- Thank you for pointing out the difference between this work and other recent efforts (Oh et al, 2017 and Pal et al, 2019), and updating the manuscript with this content. However, considering these methods have been proposed, it is only natural to ask that the proposed method needs to be compared against them, to judge the usefulness of meta-mappings as against other ways to achieve zero-shot task adaptation, at least on some common datasets/settings. The paper indeed does propose a new idea, but without a comprehensive comparison/evaluation, the inferences on its usefulness may not be conclusive without such studies.
- Considering methods solving the same/similar problem using other approaches have tested on real-world datasets, experiments on synthetic datasets alone seems limiting.  I agree and appreciate the amount of material that is already in the paper, but it is also important to have the correct material to make conclusive inferences that benefit the community.
I once again thank the authors for the efforts in the response, but just think the paper may benefit from revisions based on the comments shared here, to be more impactful/useful.