Published as a conference paper at ICLR 2018 UNSUPERVISED REPRESENTATION LEARNING BY PRE- DICTING IMAGE ROTATIONS
Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: https://github.com/gidariss/FeatureLearningRotNet.
**Paper Summary**
    This paper proposes a self-supervised method, RotNet, to learn effective image feature from images by predicting the rotation, discretized into 4 rotations of 0, 90, 180, and 270 degrees. The authors claim that this task is intuitive because a model must learn to recognize and detect relevant parts of an image (object orientation, object class) in order to determine how much an image has been rotated. 
They visualize attention maps from the first few conv layers and claim that the attend to parts of the image like faces or eyes or mouths. They also visualize filters from the first convolutional layer and show that these learned filters are more diverse than those from training the same model in a supervised manner. 
	They train RotNet to learn features of CIFAR-10 and then train, in a supervised manner, additional layers that use RotNet feature maps to perform object classification. They achieve 91.16% accuracy, outperforming other unsupervised feature learning methods. They also show that in a semi-supervised setting where only a small number of images of each category is available at training time, their method outperforms a supervised method.
	They next train RotNet on ImageNet and use the learned features for image classification on ImageNet and PASCAL VOC 2007 as well as object detection on PASCAL VOC 2007. They achieve an ImageNet and PASCAL classification score as well as an object detection score higher than other baseline methods.
    This task requires the ability to understand the types, the locations, and the poses of the objects presented in images and therefore provides a powerful surrogate supervision signal for representation learning. To demonstrate the effectiveness of the proposed method, the authors evaluate it under a variety of tasks with different settings. 
    
    
**Paper Strengths**
- The motivation of this work is well-written.
- The proposed self-supervised task is simple and intuitive. This simple idea of using image rotation to learn features, easy to implement image rotations without any artifacts
- Requiring no scale and aspect ratio image transformations, the proposed self-supervised task does not introduce any low-level visual artifacts that will lead the CNN to learn trivial features with no practical value for the visual perception tasks.
- Training the proposed model requires the same computational cost as supervised learning which is much faster than training image reconstruction based representation learning frameworks.
- The experiments show that this representation learning task can improve the performance when only a small amount of annotated examples is available  (the semi-supervised settings).
- The implementation details are included, including the way of implementing image rotations, different network architectures evaluated on different datasets, optimizers, learning rates with weight decayed, batch sizes, numbers of training epochs, etc. 
- Outperforms all baselines and achieves performance close to, but still below, fully supervised methods
- Plots rotation prediction accuracy and object recognition accuracy over time and shows that they are correlated
**Paper Weaknesses**
- The proposed method considers a set of different geometric transformations as discrete and independent classes and formulates the task as a classification task. However, the inherent relationships among geometric transformations are ignored. For example, rotating an image 90 degrees and rotating an image 180 degrees should be closer compared to rotating an image 90 degrees and rotating an image 270 degrees.
- The evaluation of low-level perception vision task is missing. In particular, evaluating the learned representations on the task of image semantic segmentation is essential in my opinion. Since we are interested in assigning the label of an object class to each pixel in the image for the task, the ability to encode semantic image feature by learning from performing the self-supervised task can be demonstrated.
- The figure presenting the visualization of the first layer filters is not clear to understand nor representative of any finding.
- ImageNet Top-1 classification results produced by Split-Brain (Zhang et al., 2016b) and Counting (Noroozi et al., 2017) are missing which are shown to be effective in the paper [Representation Learning by Learning to Count](https://arxiv.org/abs/1708.06734).
- An in-depth analysis of the correlation between the rotation prediction accuracy and the object recognition accuracy is missing. Showing both the accuracies are improved over time is not informative.
- Not fully convinced on the intuition, some objects may not have a clear direction of what should be “up” or “down” (symmetric objects like balls), in Figure 2, rotated image X^3 could plausibly be believed as 0 rotation as well, do the failure cases of rotation relate to misclassified images?
- “remarkably good performance”, “extremely good performance” – vague language choices (abstract, conclusion)
- Per class breakdown on CIFAR 10 and/or PASCAL would help understand what exactly is being learned
- In Figure 3, it would be better to show attention maps on rotated images as well as activations from other unsupervised learning methods. With this figure, it is hard to tell whether the proposed model effectively focuses on high level objects.
- In Figure 4, patterns of the convolutional filters are not clear. It would be better to make the figures clear by using grayscale images and adjusting contrast.
- In Equation 2, the objective should be maximizing the sum of losses or minimizing the negative. Also, in Equation 3, the summation should be computed over y = 1 ~ K, not i = 1 ~ N.
**Preliminary Evaluation**
This paper proposes a self-supervised task which allows a CNN to learn meaningful visual representations without requiring supervision signal. In particular, it proposes to train a CNN to recognize the rotation applied to an image, which requires the understanding the types, the locations, and the poses of the objects presented in images. The experiments demonstrate that the learned representations are meaningful and transferable to other vision tasks including object recognition and object detection. Strong quantitative results outperforming unsupervised representation learning methods, but lacking qualitative results to confirm/interpret the effectiveness of the proposed method.