Deep learning methods perform well in various tasks. However, the over-fitting problem, which causes the performance to decrease for unknown data, remains. We hence propose a method named MixFeat that directly creates latent spaces in a network that can distinguish classes. MixFeat mixes two feature maps in each latent space in the network and uses unmixed labels for learning. We discuss the difference between a method that mixes only features (MixFeat) and a method that mixes both features and labels (mixup and its family). Mixing features repeatedly is effective in expanding feature diversity, but mixing labels repeatedly makes learning difficult. MixFeat makes it possible to obtain the advantages of repeated mixing by mixing only features. We report improved results obtained using existing network models with MixFeat on CIFAR-10/100 datasets. In addition, we show that MixFeat effectively reduces the over-fitting problem even when the training dataset is small or contains errors. MixFeat is easy to implement and can be added to various network models without additional computational cost in the inference phase.
This paper follows a recent trend to improve generalization by mixing data from training samples, in this case by mixing feature maps from different samples in the latent space. One of the feature maps is added as a kind of perturbation to the other one, so only the label from the main feature map is used as the learning target. MixFeat, the proposed method of adding ‘noise’ from another learning sample is tested on CIFAR-10 and CIFAR-100 with different architectures. The authors claim that the proposed method makes the latent space more discriminative. Multiple experiments show that it helps to avoid over-fitting. 
The core idea of mixing the latent spaces of two data samples is interesting and the results seem to indicate that it improves generalization, but I have two main criticisms of this work. First, it is unclear as to why this this approach works (or why it works better than similar methods) and the explanations offered are not satisfactory. The phrase “making features discriminative in the latent space” is used repeatedly, but it is not obvious exactly what is meant by this. Design choices are also not clearly motivated, for example what is the advantage of defining a and b as was done? The second criticism is that comparisons to manifold mixup should have been included.
Approach: 
- In “1 Introduction”, the second contribution of presenting “a guideline for judging whether labels should be mixed when mixing features for an individual purpose” is not clearly communicated. 
- Figure 1 is a nice idea to illustrate the types of mixed feature distributions, but is not convincing as a toy example. A visualization of how mixed features are placed in the learned latent space for real data would be more informative. The examples showing 0.4A+0.6B and 0.6A+0.4B are confusing - it’s not clear exactly how it relates to the formulation in (1).  
- In “2.3 Computation of MixFeat” there is no clear explanation on why the authors chose a and b. Can they just be some other random small values? Is it necessary to have this correlation (cos and sin) between two feature maps we want to mix? Questions like these are not clearly explained. Similar questions can be applied to formula (4) and (6). 
+ Explicitly pointing out how backpropagation works for MixFeat in (2) (5) (7) and Figure 2 is helpful.
Experiments: 
- The authors mentioned important related works in both “1 Introduction” and “4 Relationship with Previous Work”, but in Table 1, they compared the MixFeat with only standard Mixup. Manifold Mixup would be a  better comparison as it has better performance than standard mixup and is more closely related to MixFeat - MixFeat mixes features in every latent space while Manifold Mixup does in a randomly selected space (and standard mixup only mixes the inputs). 
- The method could be described as "adding some noise along samples' latent feature directions". An interesting perspective, and would have been nice to see a comparison of MixFeat vs. perturbing with gaussian noise to see how much the direction towards other examples helps.
+ The experiments to demonstrate the effectiveness of MixFeat for avoiding over-fitting are strong (aside from the missing baseline). The experiments showing robustness to different incorrect label ratios and with different training data size are convincing.
- In Figure 6 center, the x-axis is  or ( for original MixFeat and 1D-MixFeat, and  for Inner-MixFeat), but the authors didn’t make a clear distinction in both the figure caption and “3.3.1 Dimensions and Direction of the Distribution”, having it wrong for Inner-MixFeat with “6.94% ( = 0.02)” which should be “( = 0.02)”. 
+ The ablation study motivating the choice of where to apply MixFeat was appreciated.
Related works
+ Clearly presented and covered the relevant literature. 
- It would be helpful if the differences between MixFeat and the Mixup family is more clearly stated.