  Learning interpretable disentangled representations is a crucial yet
challenging task. In this paper, we propose a weakly semi-supervised method,
termed as Dual Swap Disentangling (DSD), for disentangling using both labeled
and unlabeled data. Unlike conventional weakly supervised methods that rely on
full annotations on the group of samples, we require only limited annotations
on paired samples that indicate their shared attribute like the color. Our
model takes the form of a dual autoencoder structure. To achieve disentangling
using the labeled pairs, we follow a "encoding-swap-decoding" process, where we
first swap the parts of their encodings corresponding to the shared attribute
and then decode the obtained hybrid codes to reconstruct the original input
pairs. For unlabeled pairs, we follow the "encoding-swap-decoding" process
twice on designated encoding parts and enforce the final outputs to approximate
the input pairs. By isolating parts of the encoding and swapping them back and
forth, we impose the dimension-wise modularity and portability of the encodings
of the unlabeled samples, which implicitly encourages disentangling under the
guidance of labeled pairs. This dual swap mechanism, tailored for
semi-supervised setting, turns out to be very effective. Experiments on image
datasets from a wide domain show that our model yields state-of-the-art
disentangling performances.

Computer Vision and Pattern Recognition