COMPLEMENT OBJECTIVE TRAINING
Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the groundtruth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.
========
Summary
========
The paper deals with the training of neural networks for classification or sequence generation tasks, using a cross-entropy loss. Minimizing the cross-entropy means maximizing the predicted probabilities of the ground-truth classes (averaged over the samples). The authors introduce a "complementary entropy" loss with the goal of minimizing the predicted probabilities of the complementary (incorrect) classes. To do that, they use the average of sample-wise entropy over the complement classes. By maximizing this entropy, the predicted complementary probabilities are encouraged to be equal and therefore, the authors claim that it neutralizes them as the number of classes grows large. The proposed training procedure, named COT, consists of alternating between the optimization of the two losses.
The procedure is tested on image classification tasks with different datasets (CIFAR-10, CIFAR-100, Street View House Numbers, Tiny ImageNet and ImageNet), machine translation (training using IWSLT dataset, validation and test using TED tst2012/2013 datasets), and speech recognition (Gooogle Commands dataset). In the experiments, COT outperforms state-of-the-art models for each task/dataset.
Adversarial attacks are also considered for the classification of images of CIFAR-10: using the Fast Gradient Sign and Basic Iterative Fast Gradient Sign methods on different models, adversarial examples specifically designed for each model, are generated. Then results of these models are compared to COT on these examples. The authors admit
that the results are biased since the adversarial attacks only target part of the COT objective, hence more accurate comparisons should be done in future work.
===========================
 Main comments and questions
===========================
End of page 1: "the model behavior for classes other than the ground  truth stays unharnessed and not well-defined". The probabilities  should still sum up to 1, so if the ground truth one is maximized,  the others are actually implicitly minimized. No?
Page 3, sec 2.1: "optimizing on the complement entropy drives ŷ_ij to 1/(K − 1)". I believe that it drives each term ŷ_ij /(1 − ŷ_ig ) to be equal to 1/(K-1). Therefore, it drives ŷ_ij to (1 − ŷ_ig)/(K-1) for j!=g.
This indeed flattens the ŷ_ij for j!=g, but the effect on ŷ_ig is not controlled. In particular this latter can decrease. Then in the next step of the algorithm, ŷ_ig will be maximized, but with no explicit control over the complementary probabilities. There are two objectives that are optimized over the same variable theta. So the question is, are we sure that this procedure will converge? What prevents situations where the probabilities will alternate between two values? 
For example, with 4 classes, we look at the predicted probabilities of a given sample of class 1:
Suppose after step 1 of Algo 1, the predicted probabilities are:  0.5 0.3 0.1 0.1 
After step 2:  0.1 0.3 0.3 0.3
Then step 1: 0.5 0.3 0.1 0.1
Then step 2: 0.1 0.3 0.3 0.3
And so on... Can this happen? Or why not? Did the algorithm have trouble converging in any of the experiments?
Sec 3.1:
"additional efforts for tuning hyper-parameters might be required for optimizers to achieve the best performance": Which hyper-parameters are considered here? If it is the learning rate, why not use a different one, tuned for each objective?
Sec 3.2:
The additional optimization makes each training iteration more costly. How much more? How do the total running times of COT compare to the ones of the baselines? I think this should be mentioned in the paper.
Sec 3.4:
As the authors mention, the results are biased and so the comparison is not fair here. Therefore I wonder about the  relevance of this section. Isn't there an easy way to adapt the attacks to the two objectives to be able to illustrate the conjectured robustness of COT? For example, naively having a two steps perturbation of the input: one based on the gradient of the primary objective and then perturb the result using the gradient of the complementary objective?
===========================
Secondary comments and typos
===========================
Page 3, sec 2.1: "...the proposed COT also optimizes the complement objective for neutralizing the predicted probabilities...", using maximizes instead of optimizes would be clearer.
In the definition of the complement entropy, equation (2), C takes as parameter only y^hat_Cbar but then in the formula, ŷ_ig appears. Shouldn't C take all \hat_y as an argument in this case?
Algorithm 1 page 4: I find it confusing that the (artificial) variable that appears in the argmin (resp. argmax) is theta_{t-1}
(resp. theta'_t) which is the previous parameter. Is there a reason for this choice?
Sec 3:
"We perform extensive experiments to evaluate COT on the tasks" --> COT on tasks
"compare it with the baseline algorithms that achieve state-of-the-art in the respective domain." --> domainS
"to evaluate the model’s robustness trained by COT when attacked" needs reformulation.
"we select a state- of-the-art model that has the open-source implementation" --> an open-source implementation
Sec 3.2:
Figure 4: why is the median reported and not the mean (as in Figure 3, Tables 2 and 3)?
Table 3 and 4: why is it the validation error that is reported and not the test error?
Sec 3.3:
"Neural machine translation (NMT) has populated the use of neural sequence models": populated has not the intended meaning.
"We apply the same pre-processing steps as shown in the model" --> in the paper?
Sec 3.4:
"We believe that the models trained using COT are generalized better" --> "..using COT generalize better"
"using both FGSM and I-FGSM method" --> methodS
"The baseline models are the same as Section 3.2." --> as in Section 3.2.
"the number of iteration is set at 10." --> to 10
"using complement objective may help defend adversarial attacks." --> defend against
"Studying on COT and adversarial attacks.." --> could be better formulated
References: there are some inconsistencies (e.g.: initials versus first name)
Pros
====
- Paper is clear and well-written
- It seems to me that it is a new original idea
- Wide applicability
- Extensive convincing experimental results
Cons
====
- No theoretical guarantee that the procedure should converge
- The training time may be twice longer (to clarify)
- The adversarial section, as it is,  does not seem relevant for me