Dual learning has been successfully applied in many machine learning applications including machine translation, image-to-image transformation, etc. The high-level idea of dual learning is very intuitive: if we map an x from one domain to another and then map it back, we should recover the original x. Although its effectiveness has been empirically verified, theoretical understanding of dual learning is still missing. In this paper, we conduct a theoretical study to understand why and when dual learning can improve a mapping function. Based on the theoretical discoveries, we extend dual learning by introducing more related mappings and propose highly symmetric frameworks, cycle dual learning and multipath dual learning, in both of which we can leverage the feedback signals from additional domains to improve the qualities of the mappings. We prove that both cycle dual learning and multipath dual learning can boost the performance of standard dual learning under mild conditions. Experiments on WMT 14 English↔German and MultiUN English↔French translations verify our theoretical findings on dual learning, and the results on the translations among English, French, and Spanish of MultiUN demonstrate the efficacy of cycle dual learning and multipath dual learning.
The paper gives theorems concerning "dual learning" - that is, making
use of round-trip consistency in learning of translation and other
tasks.
There are some interesting ideas here. Unfortunately, I think there
are issues with clarity/choice of notation and correctness (errors
resulting from problems with the notation - or at least it's very
hard to figure out if things are correct under some intepretation).
More specifically, I'm uneasy about the use of x^i and x^j as defined
in section 2. In some cases x^j is a deterministic function of x^i, in
some cases it's a random variable, these cases are mixed. Section 2
becomes tangled up in this issue. It would be much better I think to
define a function f_ij(x) for each (i,j) pair that maps a sentence
x \in S^i to its correct translation f_ij(x) \in S^j.
A critical problem with the paper is that Eq. 2 is I think incorrect.
Clearly,
Pr(T_ij(x_i) = f_ij(x_i), T_ji(f_ij(x_i)) = x_i)      [1]
=
Pr(T_ij(x_i) = f_ij(x_i))                             [2]
*
Pr(T_ji(f_ij(x_i)) = x_i) | T_ij(x_i) = f_ij(x_i))    [3]
I think [1] is what is meant by the left-hand-side of Eq 2 in the paper -
though the use of x^j is ambiguous (this ambiguity is a real issue).
It can be verified that
Pr(T_ij(x_i) = f_ij(x_i)) = p_ij
however
Pr(T_ji(f_ij(x_i)) = x_i) | T_ij(x_i) = f_ij(x_i)) \neq p^r_ji
The definition of p^r_ji is that of a different quantity.
This problem unfortunately permeates the statement of theorem 1, and
the proof of theorem 1. It is probably fixable but without a
significantly revised version of the paper a reader/reviewer is
basically guessing what a corrected version of the paper would
be. Unfortunately I think publishing the paper with errors such as
this would be a problem.
Some other points:
[1] The theorem really does have to assume that there is a unique
correct translation f_ij(x^i) for each sentence x^i. Having multiple
possible translations breaks things. The authors say in section 2 "In
practice, we may select a threshold BLEU (Papineni et al., 2002a)
score, above which the translation is considered correct": this seems
to imply that the results apply when multiple translations (above
a certain BLEU score) are possible. But my understanding is that this
will completely break the results (or at least require a significant
modification of the theory).
[2] A further problem with ambiguity/notation is that T^d is never
explicitly defined. Presumably we always have T_ij^s(x^i) = T_ij(x^i) if
T_ji(T_ij(x^i)) = x^i? That needs to be explicitly stated.
[3] There may be something interesting in theorem 1 - putting aside point
[1] above - but I am just really uneasy with this theorem and its proof
given that it uses p^r_ji, and the issue with Eq. 2.
[4] Another issue with the definition of p^r_ij: the notation
P_{X^(j, r) ~ \mu}(...) where the expression ... does not refer
to X^{j, r} (instead it refers to x^j) is just really odd,
and confusing.