Federated Multi-Task Learning
Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices. In this work, we show that multi-task learning is naturally suited to handle the statistical challenges of this setting, and propose a novel systems-aware optimization method, MOCHA, that is robust to practical systems issues. Our method and theory for the first time consider issues of high communication cost, stragglers, and fault tolerance for distributed multi-task learning. The resulting method achieves significant speedups compared to alternatives in the federated setting, as we demonstrate through simulations on real-world federated datasets.
The paper generalizes an existing framework for distributed multitask learning (COCOA) and extends it to handle practical systems challenges such as communications cost, stragglers, etc. The proposed technique (MOCHA) makes the framework robust and faster.
Pros:
- Rigorous theoretical analysis
- Sound experiment methodology
Cons:
- Not very novel (compared to [47]) and just a mild variation over [47] including the analysis.
Major Comments:
---------------
1. The Equation 5 depends on \Delta \alpha_t* (minimizer of sub-problem). My understanding is that this minimizer is not known and cannot be computed accurately at every node, and therefore, we need the approximation. If so, then how do the nodes set \theta^h_t? Appendix E.3 states that this parameter is directly tuned via H_i. However, the quality should be determined by the environment, and not supposed to be 'tuned'. If tuned, then it might act as a 'learning rate' -- instead of taking the full step in the direction of the gradient, the algorithm takes a partial step.
2. The main difference between this paper and [47] seems to be that instead of the same \theta in [47], this paper has defined separate \theta^h_t for each node t.
Minor comments / typos:
-----------------------
39: ...that MTL is -as- a natural ....
108: ...of this (12) in our... -- should be (2)