  Adversarial training has achieved substantial performance in defending image
retrieval systems against adversarial examples. However, existing studies still
suffer from two major limitations: model collapse and weak adversary. This
paper addresses these two limitations by proposing collapse-oriented (COLO)
adversarial training with triplet decoupling (TRIDE). Specifically, COLO
prevents model collapse by temporally orienting the perturbation update
direction with a new collapse metric, while TRIDE yields a strong adversary by
spatially decoupling the update targets of perturbation into the anchor and the
two candidates of a triplet. Experimental results demonstrate that our
COLO-TRIDE outperforms the current state of the art by 7% on average over 10
robustness metrics and across 3 popular datasets. In addition, we identify the
fairness limitations of commonly used robustness metrics in image retrieval and
propose a new metric for more meaningful robustness evaluation. Codes will be
made publicly available on GitHub.

Computer Vision and Pattern Recognition