Multi-task learning has been successful in leveraging the dependency among different tasks to improve general performance. However, when applying to 'discrete' data (graph/text), most of the existing frameworks only leverage the general task dependency with the assumption that the task dependency remains the same for (1) different data samples; and (2) different sub-structures (node/word) in one data sample (graph/text). Yet, this assumption may not be true for real-world 'discrete' datasets. Thus, we propose a novel multi-task learning frameworkLearning To Transfer Via Modelling Multi-level Task Dependency, which leverages both general task dependency and data-specific task dependency to guide the knowledge transfer among tasks. We also propose a decomposition method to reduce the space complexity needed by transfer functions from quadratic to linear. To show the effectiveness of our framework and the importance of modelling multi-level task dependency, we conduct experiments on several public graph/text datasets, on which we obtain significant improvements over current frameworks.
The submission argues for the modeling the relationships between different tasks and incorporating such relationships when training multi-task frameworks. Though the basic concept (usefulness of modeling and incorporating the relationships among tasks) is valid, the submission has a number of critical issues, namely missing prior work that did that that already, missing critical specifics of the method, and unnecessary mix of different concepts. 
Elaborated comments: 
A) Authors seem to be unaware of critically related prior work that specially modeled task relationships and did much of what's proposed in this submission, especially "Taskonomy: Disentangling task transfer learning". The "relationship among tasks" that this submission frequently talks about is the main concept in taskonomy 2018 paper (see their abstract). Besides the apparent similarities (eg the fig 1 of this submission vs fig 1&2 of taskonomy or fig 4 of this submission vs fig 13&7 of taskonomy), the formulation has strong similarities too (transferring from "task-specific" encoders of source tasks to target tasks using transfer readout functions, or ensembling multiple task-specific representations which seem to be the same as taskonomy's higher order transfer). This submission should be majorly revised in light of prior work and the critically relevant ones should be discussed and experimentally compared to. 
B) The presentation suffers from missing critical specifics. For instance, the "general task dependency" matrix  shown in Fig 3 and mentioned in page 4, which seem to be the same concept as taskonomy's task affinity matrix, is only mentioned in passing. While that seem to be one of the most important components of the method and its definition and extraction method should be discussed. 
C) Inline with the point B above, the presentation of the "Transfer Block" and what the authors refer to as "Point-wise Mutual Attention Mechanism" has issues and missing details. This block could potentially have new points in it, but it's not feasible to judge that and its technical correctness given the current disposition. For instance eq 2 seem to suggest the authors develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from (to reduce T^2 complexity to 2T). The rest of the section does not provide a clear implementation of this and add mathematical/notation confusions. Eg H_i_j is defined to be the task-specific representation of the source task i but is indexed over both tasks i and j where j is the target, or there is a E_j(X_j) where both indexes are j while E's index is over tasks and X's index is over datapoints. 
Similarly the submission seem to jump over certain concepts/terms e.g. "multi-view task dependency" in page 4 vs"multi-level task dependency" in the title, etc. What exactly "view" or "level" mean here? Are those phrases really needed? Dropping any loosely grounded phrase would be a useful practice toward a clearer presentation.
Overall, unfortunately the submission suffers from serious issues in its current shape. 
----
Comments after rebuttal stage: 
Thanks to the authors for the rebuttal. It provided some help, but unfortunately it doesn't resolve the majority of the issues, as most of them are too major. A clear discussion on how the proposal is different and why it is better than the recent works that were not cited would be needed, and likely authors needed strong experimental comparison with some of them, eg to prove both general and data-specific task dependency is needed. 
I also didn't find the hierarchical justification clear or convincing "The reason is that different from the image which [1] focuses on, text and graph data are hierarchical: word -> sentence and node -> graph. The task dependency at the basic level (word and node) may be different from the general task dependency".