We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark.
### Summary:
This paper propose a new model for learning generic feature representations for visual-linguistic tasks by pretraining on large-scale vision and language datasets like Conceptual Captions and language-only datasets like BookCorpus and English Wikipedia. They demonstrate that the pre-training procedure can help improve performance on down-streaming tasks like visual question answering, visual commonsense reasoning. 
Overall I liked the design choices made in the presentation. Although the paper doesn't provide insights around what the representations have learned and how they differ from representations learned / used by existing methods, they have provided substantial evidence to suggest that pre-training helps in a lot of downstream tasks. 
Although it's hard to evaluate the paper without putting it in context with other concurrent works that have come out recently, I tried my best to evaluate the merits of the paper in isolation. 
### Strengths:
- The paper explores an interesting direction of learning generic feature representations for visual-linguistic tasks for down-streaming tasks. Traditionally, people learn feature representations from scratch for each downstream task which might not always be possible if the training data is limited.
- The paper does a decent job mentioning all the concurrent work in the space of learning multi-modal representations that have come out very recently. They distinguish the proposed method from existing work and also compare the performance of the proposed approach with concurrent work on downstream tasks showing performance on-par or better than existing methods.
- I liked some of the design choices made in the paper. (1)  Instead of training a separate  transformer network for each type of input segments (question, ans, caption, image, etc). This makes the model easily extensible to other tasks as long as the correct segment embeddings are used to identify different input sources. (2) They also use a separate embedding for visual features instead of a common embedding  for both language tokens and visual tokens.
- Unlike the pre-training task in concurrent work, the model was pre-trained not just on multi-modal datasets like conceptual captions but also on text-only corpus like BookCorpus and English Wikipedia. The authors claim that this leads them to learn better representations for longer sentences which they found useful for VCR task.
### Weaknesses:
- The authors claim that attention mechanism in cross-modal transformer by concurrent approaches is restrictive but doesn't give substantial evidence that this is true. What are the limitations for cross-modal attention mechanisms compared to a single transformer model as described in this paper.
- On the contrary, by having a cross modal architecture, they can pre-train each modality separately on unaligned data. For instance, the text only transformer can be trained using large text corpora similar to BERT while the image only transformer can be trained on big datasets like OpenImages, ImageNet etc
- While the paper gives a lot of empirical evidence that pre-training helps, it would have been interesting to develop an understanding of what the model is actually learning and how are these representations better than learning representations from scratch for each task. For instance maybe the authors can visualize attention similar to [1].
### Other questions:
- When training on text-only datasets, what is the input on Visual Feature Embedding since there are no associated images. The authors mention that for non-visual elements, the features are extracted on the whole image. It's still unclear what the associated visual features are for text-only datasets.
- One of the pre-training tasks is masked ROI classification but it assigns a hard label to each ROI feature. It might be interesting to instead try learn the entire probability distribution (the output of a pre-trained classifier) by either minimizing the KL-divergence or by using softmax with soft-targets.
- While the model was learnt on text-only data, as mentioned in the above section, will the model help from image-only datasets such as large-scale classification datasets?
- While the models are tested on vision-and-language datasets, will these generic representations also be useful for unimodal tasks?