summaries: This paper aims to incorporate world knowledge for the pretraining approach so that (1) pretrained models contain useful information about the world, and (2) benefit downstream NLP tasks. The paper does so by introducing the objective which distinguishes the groundtruth entity and the false entity in the Wikipedia text. This was carefully done by detecting entities in the text, find the corresponding entity in Wikidata, randomly choose another entity which has the same type as the original entity, and make sure this doesn’t happen for neighboring entities or too much in order to avoid context change. Adding this objective to the original masked LM objective, this pretrained model is shown to be effective and outperform baselines significantly in many tasks such as zero-shot fact completion, question answering, and fine-grained entity typing.

summaries: This paper proposes to trained better entity centric text embeddings by switching entities mentioned in the text to some other entities with the same type. The target is modeled as a binary classification task, which is trained jointly with the MLM loss. The authors do experiments on multiple tasks, and the model shows strong performance on all tasks. And the ablation study justifies that "knowledge pre-training" is crucial. The idea is novel and the experiment results suggest that the additional "adversarial" target helps. The writing is clear in general, but misses some implementation details.

summaries: This paper proposed to improve pre-training of language models (e.g. BERT) by incorporating information around entities based on English Wikipedia. The idea is very simple and straightforward: it takes all the anchor links from Wikipedia and replaces some entities by randomly sampling negative ones of the same entity type (according to Wikidata) and adds an extra binary prediction task which predicts if the entity has been replaced or not.

abstractText: Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

authors:
Sungjin Ahn
Heeyoul Choi
Tanel Pärnamaa
Yoshua Bengio

title: A neural knowledge language model

venue: arXiv preprint arXiv:1608.00318,

authors:
Jonathan Berant
Andrew Chou
Roy Frostig
Percy Liang

title: Semantic parsing on freebase from question-answer pairs

venue: In EMNLP,

authors:
Antoine Bordes
Nicolas Usunier
Alberto Garcı́a-Durán
Jason Weston
Oksana Yakhnenko

title: Translating embeddings for modeling multi-relational data

venue: In NIPS,

authors:
Kevin Clark
Urvashi Khandelwal
Omer Levy
Christopher D Manning

title: What does bert look at? an analysis of bert’s attention

authors:
Bhuwan Dhingra
Kathryn Mazaitis
William W Cohen

title: Quasar: Datasets for question answering by search and reading

venue: arXiv preprint arXiv:1707.03904,

authors:
Matthew Dunn
Levent Sagun
Mike Higgins
V Ugur Guney
Volkan Cirik
Kyunghyun Cho

title: Searchqa: A new q&a dataset augmented with context from a search engine

venue: arXiv preprint arXiv:1704.05179,

title: Frame semantics and the nature of language

venue: In Annals of the New York Academy of Sciences: Conference on the origin and development of language and speech,

authors:
Karl Moritz Hermann
Tomás Kociský
Edward Grefenstette
Lasse Espeholt
Will Kay
Mustafa Suleyman
Phil Blunsom

title: Teaching machines to read and comprehend

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Xiao Ling
Daniel S. Weld

title: Fine-grained entity recognition

venue: In AAAI. AAAI Press,

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

venue: arXiv preprint arXiv:1907.11692,

authors:
Robert Logan
Nelson F. Liu
Matthew E. Peters
Matt Gardner
Sameer Singh

title: Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling

venue: In ACL,

authors:
Bryan McCann
James Bradbury
Caiming Xiong
Richard Socher

title: Learned in translation: Contextualized word vectors

authors:
Todor Mihaylov
Anette Frank

title: Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

authors:
Tomas Mikolov
Ilya Sutskever
Kai Chen
Greg S Corrado
Jeff Dean

title: Distributed representations of words and phrases and their compositionality

authors:
Maximilian Nickel
Volker Tresp
Hans-Peter Kriegel

title: A three-way model for collective learning on multi-relational data

venue: In ICML,

authors:
Myle Ott
Sergey Edunov
Alexei Baevski
Angela Fan
Sam Gross
Nathan Ng
David Grangier
Michael Auli

title: fairseq: A fast, extensible toolkit for sequence modeling

venue: In Proceedings of NAACL-HLT 2019: Demonstrations,

authors:
Jeffrey Pennington
Richard Socher
Christopher D. Manning

title: Glove: Global vectors for word representation

authors:
Matthew E Peters
Mark Neumann
Mohit Iyyer
Matt Gardner
Christopher Clark
Kenton Lee
Luke Zettlemoyer

title: Deep contextualized word representations

venue: arXiv preprint arXiv:1802.05365,

authors:
Matthew E. Peters
Mark Neumann
Luke Zettlemoyer
Wen-tau Yih

title: Dissecting contextual word embeddings: Architecture and representation

authors:
Matthew E Peters
Mark Neumann
IV Logan
L Robert
Roy Schwartz
Vidur Joshi
Sameer Singh
Noah A Smith

title: Knowledge enhanced contextual word representations

authors:
Fabio Petroni
Tim Rocktäschel
Patrick Lewis
Anton Bakhtin
Yuxiang Wu
Alexander H. Miller
Sebastian Riedel

title: Language models as knowledge bases

authors:
Alec Radford
Jeffrey Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI Blog,

authors:
Pranav Rajpurkar
Jian Zhang
Konstantin Lopyrev
Percy Liang

title: Squad: 100, 000+ questions for machine comprehension of text

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Denny Vrandečić
Markus Krötzsch

title: Wikidata: a free collaborative knowledge base

authors:
Alex Wang
Amanpreet Singh
Julian Michael
Felix Hill
Omer Levy
Samuel R. Bowman

title: GLUE: A multi-task benchmark and analysis platform for natural language understanding

venue: In ICLR,

authors:
Shuohang Wang
Mo Yu
Xiaoxiao Guo
Zhiguo Wang
Tim Klinger
Wei Zhang
Shiyu Chang
Gerry Tesauro
Bowen Zhou
Jing Jiang

title: Reinforced ranker-reader for open-domain question answering

venue: In AAAI,

authors:
Shuohang Wang
Mo Yu
Jing Jiang
Wei Zhang
Xiaoxiao Guo
Shiyu Chang
Zhiguo Wang
Tim Klinger
Gerald Tesauro
Murray Campbell

title: Evidence aggregation for answer re-ranking in open-domain question answering

authors:
Zhiguo Wang
Patrick Ng
Xiaofei Ma
Ramesh Nallapati
Bing Xiang

title: Multi-passage bert: A globally normalized bert model for open-domain question answering

venue: arXiv preprint arXiv:1908.08167,

authors:
Wenhan Xiong
Thien Hoang
William Yang Wang

title: DeepPath: A reinforcement learning method for knowledge graph reasoning

venue: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,

authors:
Wenhan Xiong
Mo Yu
Shiyu Chang
Xiaoxiao Guo
William Yang Wang

title: Improving question answering over incomplete KBs with knowledge-aware reader

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Wei Yang
Yuqing Xie
Aileen Lin
Xingyu Li
Luchen Tan
Kun Xiong
Ming Li
Jimmy Lin

title: End-to-end open-domain question answering with bertserini

venue: arXiv preprint arXiv:1902.01718,

authors:
Wei Yang
Yuqing Xie
Luchen Tan
Kun Xiong
Ming Li
Jimmy Lin

title: Data augmentation for bert fine-tuning in open-domain question answering

venue: arXiv preprint arXiv:1904.06652,

venue: arXiv preprint arXiv:1906.08237,

authors:
Rowan Zellers
Yonatan Bisk
Roy Schwartz
Yejin Choi

title: SWAG: A large-scale adversarial dataset for grounded commonsense inference

authors:
Sheng Zhang
Xiaodong Liu
Jingjing Liu
Jianfeng Gao
Kevin Duh
Benjamin Van Durme

title: Record: Bridging the gap between human and machine commonsense reading comprehension

venue: arXiv preprint arXiv:1810.12885,

authors:
Yukun Zhu
Ryan Kiros
Richard S. Zemel
Ruslan Salakhutdinov
Raquel Urtasun
Antonio Torralba
Sanja Fidler

title: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

venue: In ICCV,

heading: 1 INTRODUCTION

text:
Language models pretrained on a large amount of text such as ELMo (Peters et al., 2018a)), BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019c) have established new state of the art on a wide variety of NLP tasks. Researchers ascertain that pretraining allows models to learn syntactic and semantic information of language that is then transferred on other tasks (Peters et al., 2018b; Clark et al., 2019). Interestingly, pretrained models also perform well on tasks that require grounding language and reasoning about the real world. For instance, the new state-of-the-art for WNLI (Wang et al., 2019a), ReCoRD (Zhang et al., 2018) and SWAG (Zellers et al., 2018) is achieved by pretrained models. These tasks are carefully designed so that the text input alone does not convey the complete information for accurate predictions – external knowledge is required to fill the gap. These results suggest that large-scale pretrained models implicitly capture real-world knowledge. Logan et al. (2019) and Petroni et al. (2019) further validate this hypothesis through a zero-shot fact completion task that involves single-token entities, showing that pretrained models achieve much better performance than random guessing and can be on par with specifically-trained relation extraction models.
As unstructured text encodes a great deal of information about the world, large-scale pretraining over text data holds the promise of simultaneously learning syntax, semantics and connecting them with knowledge about the real world within a single model. However, existing pretraining objectives are usually defined at the token level and do not explicitly model entity-centric knowledge. In this work, we investigate whether we can further enforce pretrained models to focus on encyclopedic knowledge about real-world entities, so that they can better capture entity information from natural language and be applied to improving entity-related NLP tasks. We evaluate the extent to which a pretrained model represents such knowledge by extending an existing fact completion evaluation to a cloze ranking setting that allows us to deal with a large number of multi-token entity names without manual judgments. Our experiments on 10 common Wikidata (Vrandečić & Krötzsch, 2014) relations reveal that existing pretrained models encode entity-level knowledge only to a limited degree. Thus, we propose a new weakly supervised knowledge learning objective that requires the
model to distinguish between true and false knowledge expressed in natural language. Specifically, we replace entity mentions in the original documents with names of other entities of the same type and train the models to distinguish the correct entity mention from randomly chosen ones. Models trained with this objective demonstrates much stronger fact completion performance for most relations we test on. Compared with previous work (Zhang et al., 2019; Peters et al., 2019) that utilizes an external knowledge base to incorporate entity knowledge, our method is able to directly derive real-world knowledge from unstructured text. Moreover, our method requires no additional data processing, memory or modifications to the BERT model when fine-tuning for downstream tasks.
We test our model on two practical NLP problems that require entity knowledge: Question Answering (QA) and fine-grained Entity Typing. We use four previously published datasets for open-domain QA and observe that questions in these datasets often concern entities. The Entity Typing task requires the model to recognize fine-grained types of specified entity mentions given short contexts. On three of the QA datasets, our pretrained model outperforms all previous methods that do not rely on memory-consuming inter-passage normalizations1. On the FIGER entity-typing dataset, our model sets a new state of the art. Through ablation analysis, we show that the new entity-centric training objective is instrumental for achieving state-of-the-art results.
In summary, this paper makes the following contributions: 1) We extend existing fact completion evaluation settings to test pretrained models’ ability on encoding knowledge of common real-world entities; 2) We propose a new weakly supervised pretraining method which results in models that better capture knowledge about real-world entities from natural language text; 3) The model trained with our knowledge learning objective establishes new state of the art on three entity-related QA datasets and a standard fine-grained entity typing dataset.
We begin by introducing our weakly supervised method for knowledge learning (§2) and then discuss experiment settings and evaluation protocols, compare our model to previously published work and perform ablation analysis. Finally, we review related work in §4 and conclude in §5.

heading: 2 ENTITY REPLACEMENT TRAINING

text:
We design an entity-centric training objective that utilizes weakly supervised training signals to explicitly encourage knowledge learning during pretraining. Given an input document, we first
1Wang et al. (2019b) propose to use BERT to encode multiple retrieved paragraphs within the same forwardbackward pass. This operation requires around 12G GPU memory even for encoding a single example which includes 6 512-token paragraphs.
recognize the entity mentions and link them to Wikipedia entities2. We consider the original texts as positive knowledge statements and create negative statements by randomly replacing the entity mentions (E+) with the names of other random entities (E−) that have the same entity type as the mentioned entity. This setup is similar in spirit to the type-constrained negative sampling technique used to train knowledge base representations (Bordes et al., 2013). The latter technique creates negative triples by replacing the subject or object entity with random entities of the same type. Instead of knowledge base triples, we treat unstructured texts as factual statements. For a certain entity e mentioned in a context C, we train the model to make a binary prediction indicating whether the entity has been replaced:
Je,C = 1e∈E+ logP (e|C) + (1− 1e∈E+) log(1− P (e|C)).
Compared to the language modeling objective, entity replacement is defined at the entity level and introduces stronger negative signals. When we enforce entities to be of the same type, we preserve the linguistic correctness of the original sentence while the system needs to learn to perform judgment based on the factual aspect of the sentence.
We describe the implementation in more detail in the following paragraphs.
Data Preparation We use the whole English Wikipedia dump as training data and rely on all Wikipedia entities3. Entities in documents are recognized based on Wikipedia anchor links and entity alias from Wikidata. That is, we first retrieve the entities annotated by anchor links and then find other mentions of these entities by string matching their Wikidata alias. We split each document into multiple text chunks with the same size (512 tokens). Although our experiments rely on the Wikipedia corpus, this setup can be easily extended to larger corpora with off-the-shelf entity linking tools. We leave the larger scope of the experiments to future work.
Replacement Strategy When replacing entities, we first lookup type information4 from Wikidata and then randomly select other entities with the same type. We do not replace adjacent entities. In other words, there must be at least one unreplaced entity between any two replaced ones. This reduces cases where we replace all entities in the same sentence and the resulting sentences happen to introduce correct entities by chance. For replacement, we randomly sample a string from the entities’ alias set. For each text chunk, we replicate it 10 times with different negative entities for each replacement location. We show an illustration of the entity replacement method in Figure 1.
Model Architecture We use the Transformer (Vaswani et al., 2017) model used by BERT (Devlin et al., 2019). We use the same architecture as BERT base: 12 Transformer layers, each with hidden dimension 768. We initialize the transformer with a model pretrained based on our own BERT reimplementations5. For each entity, we use the final representations of its boundary words (words before and after the entity mention) to make predictions. We simply concatenate the boundary words’ representations and add a linear layer for prediction. During training, we use 0.05 dropout at the final layer.
Training Objectives Masked language model pretraining has been proven to be effective for downstream tasks. While training for entity replacement we also train with the masked language model objective in a multi-task set-up. When masking tokens, we restrict the masks to be outside the entity spans. We use a masking ratio of 5% instead of 15% in the original BERT to avoid masking out too much of the context. We train the model for approximately 1 million updates using a batch size of 128.
2The entity links are only required for pretraining. 3Each Wikipedia entity (title) corresponds to a unique entity node in Wikidata. 4Based on the “instance of” relation in Wikidata. If there are multiple correct types, we randomly choose one. 5We use the masked language model implementation in Fairseq (Ott et al., 2019) to pre-train model for 2M updates on the combination of BooksCorpus (Zhu et al., 2015) and English Wikipedia.

heading: 3 EXPERIMENTS

text: We first test our model on a fact completion task. This task resembles traditional knowledge base completion: it requires the model to complete missing entities in factual triples. We further test on two real-world downstream tasks that require entity-level knowledge – question answering and fine-grained entity typing. We describe the hyperparameter and training settings of all experiments in the appendix.

heading: 3.1 ZERO-SHOT FACT COMPLETION

text:
In traditional knowledge base completion tasks models have access to a set of training triples. Instead, we utilize a zero-shot test to examine the model’s ability to automatically derive relational knowledge from natural language.
Dataset We rely on factual triples from Wikidata. Each triple describes the relationship between two certain entities, e.g., {Paris, CapitalOf, France}. Following recent practices (Bosselut et al., 2019; Logan et al., 2019) that decode structured knowledge from language models, we first manually create templates to convert triples of 10 common relations into natural language expressions ({Paris, CapitalOf, France}→ the capital of France is Paris). We then create queries by removing the object entity in the expression and use pre-trained models to predict the missing entities, e.g., the capital of France is ?. We create 1000 cloze examples6 for each of the 10 relations.
Evaluation Metrics Previous work (Logan et al., 2019; Petroni et al., 2019) either relies on human evaluation or only considers single-token entities for fact completion. In contrast, we consider an entity-ranking setup and create a set of candidate entities for each relation. This setting allows us to automatically evaluate a large number of queries that usually involve multi-token entities. We test pretrained models on their ability to recover the correct object entity from the candidate set. To create the negative choices, we select from the set of all object entities in the particular relation, which generally have the same type as the groundtruth and are more challenging to distinguish than entities with different types. Our evaluation strategy is similar to previous work on knowledge base completion (Nickel et al., 2011; Bordes et al., 2013; Xiong et al., 2017). We follow these studies and use Hits@10 as the evaluation metric.
Baselines We compare our model with two pretrained language models BERT (Devlin et al., 2019) (both base and large) and GPT-2 (Radford et al., 2019). We make use of their output token probabilities to rank candidate entities. For BERT, we feed in the masked queries (e.g., Qmasked = the capital of France is [MASK]). For multi-token candidates, we use the same number of [MASK] tokens in the query inputs. We use the average log probability of masked tokens for ranking. Given a multi-token entity Ei = [e1i , e 2 i , ..., e |Ei| i ], the ranking score from BERT is calculated as
SEi = 1 |Ei| ∑ k logP (eki |Qmasked).
For GPT-2, we feed in the original query without the answer entity and use the first-token probability of candidate entities for ranking, which performs better than using average log probabilities. As our model learns to predict a plausible probability (P (e|C)) for each entity mention during entity replacement training, we can directly use these predicted probabilities to rank the candidates.
Results Table 1 shows the fact completion results for all relations. We denote our method WKLM for (Weakly Supervised Knowledge-Pretrained Language Model). Overall, WKLM achieves the best results on 8 of the 10 relations. We also observe that GPT-2 outperforms BERT on average. We think this is because the fact completion task requires models to predict the missing entities using only a short context on the left, while BERT pretraining incorporates context from both directions. Interestingly, BERT achieves good performance on several geographical relations such as PlaceOfBirth, LocatedIn and PlaceOfDeath. We conjecture that this is because location entities usually appear at sentence ends in Wikipedia articles, e.g., Obama was born in
6For each relation, we use the top triples that connecting most common entities.
Honolulu, Hawaii.. This sentence pattern is similar to our templates and BERT may learn to rely mostly on the left context to make predictions. For most relations that include answers that are person names, BERT lags behind both GPT-2 and our model.
Comparing the top and bottom five relations, we observe that BERT’s performance is correlated with the size of the candidate set, while WKLM and GPT-2 are less sensitive to this number. A similar pattern exists between models’ performance and the cardinality of groundtruth answers, i.e., our model achieves similar performance on both single-answer and multiple-answer queries while BERT is usually better at single-answer queries. WKLM both outperforms BERT and GPT-2 and achieves robust performance across relations with different properties. Visualization of correlations between relation properties and model performance can be found in the appendix.

heading: 3.2 DOWNSTREAM TASKS

text: Background knowledge is important for language understanding. We expect our pretraining approach to be beneficial to NLP applications where entity-level knowledge is essential. We consider two such applications: question answering and entity-typing. We find that a large portion of the questions in existing QA datasets are about entities and involve entity relations. In a way, our pretraining objective is analogous to question answering in a multiple-choice setting (Hermann et al., 2015). The entity-typing task requires the model to predict a set of correct types of entity mentions in a short context. The context itself can be insufficient and the training data for this task is small and noisy. We believe a model that encodes background entity knowledge can help in both cases.

heading: 3.2.1 QUESTION ANSWERING

text:
Datasets We consider four question answering datasets:
• WebQuestions (Berant et al., 2013) is originally a dataset for knowledge base question answering. The questions are collected using Google Suggest API and are all asking about simple relational facts of Freebase entities. • TriviaQA7 (Joshi et al., 2017) includes questions from trivia and quiz-league websites.
Apart from a small portion of questions to which the answers are numbers and free texts, 92.85% of the answers are Wikipedia entities. • Quasar-T (Dhingra et al., 2017) is another dataset that includes trivia questions. Most of
the answers in this dataset are none phrases. According to our manual analysis on random samples, 88% of the answers are real-world entities8. • SearchQA (Dunn et al., 2017) uses questions from the television quiz show Jeopardy! and
we also find that almost all of the answers are real-world entities. 7The splits of TriviaQA might be different in previous work, we use the same splits used by Lin et al. (2018). Different methods also use different retrieval systems for this dataset, i.e., ORQA and BERTserini retrieve documents from Wikipedia while the other methods use retrieved Web documents.
8We consider answers as entities as long as they correspond to Wikidata entities or Wikipedia titles.
Questions in all three datasets are created without the context of a paragraph, which resembles the scenario of practical question answering applications. All the questions except WebQuestions are written by humans. This indicates that humans are generally interested to ask questions to seek information about entities. We show the statistics and example questions in Table 2. We split the training data (created by distant supervision) of WebQuestions with a ratio (9:1) for training and development. Since our model is based on our own BERT implementations, in addition to the aforementioned entity-related datasets, we first use the standard SQuAD (Rajpurkar et al., 2016) benchmark to validate our model’s answer extraction performance.
Settings We adopt the fine-tuning approach to extract answer spans with pretrained models. We add linear layers over the last hidden states of the pretrained models to predict the start and end positions of the answer. Unlike SQuAD, questions in the datasets we use are not paired with paragraphs that contain the answer. We follow previous work (Chen et al., 2017; Wang et al., 2018a) and retrieve context paragraphs with information retrieval systems. Details of the context retrieval process for each dataset can be found in the appendix. Reader models are trained with distantly supervised data, i.e., we treat any text span in any retrieved paragraph as ground truth as long as it matches the original answers. Since the reader model needs to read multiple paragraphs to predict a single answer at inference time, we also train a BERT based paragraph ranker with distant-supervised data to assign each paragraph a relevance score. The paragraph ranker takes question and paragraph pairs and predicts a score in the range [0, 1] for each pair. During inference, for each question and its evidence paragraph set, we first use the paragraph reader to extract the best answer from each paragraph. These answers are then ranked based on a linear combination of the answer extraction score (a log sum of the answer start and end scores) and the paragraph relevance score. We also evaluate model performance without using the relevance scores.
Open-Domain QA Baselines We compare our QA model with the following systems:
• DrQA (Chen et al., 2017) is an open-domain QA system which uses TF-IDF with bigram features for ranking and a simple attentive reader for answer extraction.
• R3 (Wang et al., 2018a) is a reinforcement learning based system which jointly trains a paragraph ranker and a document reader.
• DSQA (Lin et al., 2018) uses RNN-based paragraph ranker and jointly trains the paragraph ranker and attentive paragraph ranker with a multi-task loss.
• Evidence Aggregation (Wang et al., 2018b) uses a hybrid answer reranking module to aggregate answer information from multiple paragraphs and rerank the answers extracted from multiple paragraphs.
• BERTserini (Yang et al., 2019a) is a BERT-based open-domain QA system, which uses BM25-based retriever to retrieve 100 paragraphs and a BERT-based reader to extract answers. The paragraph reader is either trained with SQuAD (Rajpurkar et al., 2016) data or distant-supervision data (Yang et al., 2019b)
• ORQA (Lee et al., 2019) replaces the traditional BM25 ranking with a BERT-based ranker. The ranker model is pretrained on the whole Wikipedia corpus with an inverse cloze task which simulates the matching between questions and paragraphs. All text blocks in Wikipedia are be pre-encoded as vectors and retrieved with Locality Sensitive Hashing.
Results Table 3 shows the SQuAD results and Table 4 shows the open-domain results on the four datasets that are highly entity-related. From the SQuAD results, we observe that our BERT reimplementation performs better than the original model this is due to the fact that it is trained
for twice as many updates: 2 million vs. 1 million for the original BERT. Although lots of the answers in SQuAD are non-entity spans, the WKLM model we propose achieves better performance than BERT. We believe the improvement is due to both the masked language model and entity replacement objectives. Ablation experiments on the training objectives will be discussed in §3.2.3.
Having established that our BERT re-implementation performs better than the original model, we compare with only our own BERT for the following experiments. From Table 4, we see that our model produces consistent improvements across different datasets. Compared to the 0.8 F1 improvements over BERT on SQuAD, we achieve an average of 2.7 F1 improvements over BERT on entity-related datasets when the ranking scores are not used. On TriviaQA and Quasar-T, WKLM outperforms our BERT even
when it uses ranking scores. Improvements in natural language question datasets (WebQuestions, TriviaQA, and Quasar-T) are more significant than SearchQA where the questions are informal queries. When we utilize ranking scores from a simple BERT based ranker, we are able to achieve the state-of-the-art on three of the four datasets.

heading: 3.2.2 ENTITY TYPING

text:
To compare with an existing study (Zhang et al., 2019) that also attempts to incorporate entity knowledge into language models, we consider an additional entity typing task using the large FIGER dataset (Ling & Weld, 2012). The task is to assign a fine-grained type to entity mentions. We do that by adding two special tokens before and after the entity span to mark the entity position. We use the final representation of the start token ([CLS]) to predict the entity types. The model is fine-tuned on weakly-supervised training data with binary cross-entropy loss. We evaluate the models using strict accuracy, loose micro, and macro F1 scores.
We show the results in Table 5. We compare our model with two non-BERT neural baselines (Inui et al., 2017) that integrate a set of hand-crafted features: LSTM + Hand-crafted and Attentive +
Hand-crafted; a vanilla BERT baseline and the ERNIE model (Zhang et al., 2019) that enhances BERT with knowledge base embeddings.
First, we see that naively applying BERT is less effective than simple models combined with sparse hand-crafted features. Although the ERNIE model can improve over BERT by 5.15 points, its performance still lags behind models that make good use of hand-crafted features. In contrast, although based on a stronger BERT model, our model achieves larger absolute improvements (5.68 points) and sets a new state-of-the-art for this task. Given the larger improvement margin, we believe our model that directly learn knowledge from text is more effective than the ERNIE method.

heading: 3.2.3 ABLATION STUDY: THE EFFECT OF MASKED LANGUAGE MODEL LOSS

text:
In view of a recent study (Liu et al., 2019b) showing simply extending the training time of BERT leads to stronger performance on various downstream tasks, we conduct further analysis to differentiate the effects of entity replacement training and masked language modeling. We compare our model with three variants: a model pretrained only with the knowledge learning objective (WKLM without MLM), a model trained with both knowledge learning and masked language modeling with more masked words (WKLM with 15% MLM) and a BERT model trained with additional 1 million updates on English Wikipedia (BERT + 1M MLM updates) and no knowledge learning.
The ablation results are shown in Table 6. The results of WKLM without MLM validate that adding the language model objective is essential for downstream performance. We also find that masking out too many words (i.e., 15% masking ratio as in the original BERT) leads to worse results. We conjecture that too many masked words outside entity mentions break parts of the context information and introduce noisy signals to knowledge learning. Results of continued BERT training show that more MLM updates are often beneficial, especially for SQuAD. However, on tasks that are more entity-centric, continued MLM training is less effective than our WKLM method. This suggests that our WKLM method could serve as an effective complementary recipe to masked language modeling when applied to entity-related NLP tasks.

heading: 4 RELATED WORK

text:
Pretrained Language Representations Early research on language representations focused on static unsupervised word representations (Mikolov et al., 2013; Pennington et al., 2014). Word embeddings leverage co-occurrences to learn latent word vectors that approximately reflect word semantics. Given that words can have different meanings in different contexts, more recent studies (McCann et al., 2017; Peters et al., 2018a) show that contextual language representations can be more powerful than static word embeddings in downstream tasks. This direction has been further explored at a larger scale with efficient Transformer architectures (Radford et al., 2019; Devlin et al., 2019; Yang et al., 2019c). Our WKLM method is based on these techniques and we focus on improving the knowledge ability of pretrained models.
Knowledge-Enhanced NLP Models Background knowledge has been considered an indispensable part of language understanding (Fillmore et al., 1976; Minsky, 1988). As standard language encoders usually do not explicitly model knowledge, recent studies (Ahn et al., 2016; Yang & Mitchell, 2017; Logan et al., 2019; Liu et al., 2019a) have explored methods to incorporate external knowledge into NLP models. Most of these methods rely on additional inputs such as entity representations from structured knowledge bases. With the breakthrough of large-scale pretrained language
encoders (Devlin et al., 2019), Zhang et al. (2019) and Peters et al. (2019) adopt similar ideas and propose entity-level knowledge enhancement training objectives to incorporate knowledge into pretrained models. Other recent studies (Mihaylov & Frank, 2018; Xiong et al., 2019) leverage external knowledge bases to enhance text-based question answering models. In contrast to these methods, our method utilizes minimal external entity information and does not require additional memory or architectural changes when applied to downstream tasks.

heading: 5 CONCLUSION

text: We introduce a weakly supervised method to encourage pretrained language models to learn entitylevel knowledge. Our method uses minimal entity information during pretraining and does not introduce additional computation, memory or architectural overhead for downstream task fine-tuning. The trained model demonstrates strong performance on a probing fact completion task and two entity-related NLP tasks. Together, our results show the potential of directly learning entity-level knowledge from unstructured natural language and the benefits of large-scale knowledge-aware pretraining for downstream NLP tasks.

heading: A APPENDIX

text:
Implementation Details and Hyperparameters We implement our method using Fairseq Ott et al. (2019) and the fact completion baselines are implemented with Huggingface’s PytorchTransformers9. We pretrain the models with 32 V100 GPUs for 3 days. We use at most 2 GPUs for fine-tuning the paragraph reader, use 8 GPUs for fine-tuning the paragraph ranker. The entity-typing experiments require larger batch sizes and take 8 GPUs for training.
For the knowledge learning pretraining phase, we use the Adam optimizer (Kingma & Ba, 2014) with learning rate 1e-5, batch size 128 and weight decay 0.01. The model is pretrained on 32 V100 GPUs for 3 days. To train the paragraph reader for open-domain QA, we select the best learning rate from {1e-6, 5e-6, 1e-5, 2e-5} and last layer dropout ratio from {0.1, 0.2}. We set the maximum training epoch to be 10 and batch size to be 32. The maximal input sequence length is 512 for WebQuestions and 128 for the other three datasets that use sentence-level paragraphs. For the paragraph ranker, we choose learning rate from {1e-5, 2e-5, 5e-6}, use dropout 0.1 and batch size 256. The maximal sequence length for each dataset is consistent with the one we used for training the paragraph reader. The linear combination of ranking and extraction scores is selected based on validation performance. For SQuAD experiments, we select learning rate from {1e-5, 5e-6, 2e-5, 3e-5}, learning rate from {8, 16}, last layer dropout ratio from {0.1, 0.2}. We set the maximal sequence length as 512 and the maximal training epoch as 5. For entity typing, we select learning rate from {1e-5, 2e-5, 3e-5, 5e-5} and batch size from {128, 256}. We set the maximal sequence length to be 256, the last layer dropout ratio to be 0.1. The model is fine-tuned for at most 3 epochs to prevent overfitting. The threshold for type prediction is selected on the validation set.
Context Collection for QA Datasets For WebQuestions, we collect evidence context using the document retriever of DrQA (Chen et al., 2017), which uses TF-IDF based metric to retrieve the top 5 Wikipedia articles. For Quasar-T, we use Lucene ranked paragraphs. For SearchQA and TriviaQA, we use paragraphs ranked by search engines. Following existing research (Wang et al., 2018b; Lin et al., 2018), we use sentence-level paragraphs for SearchQA (50 sentences), TriviaQA (100 sentences) and SearchQA (100 sentences).
Correlation between Fact Completion Results and Properties of Relations Figure 2 shows the fact completion results of BERT are unstable on different relations with different properties, i.e., BERT’s performance is strongly correlated with the size of candidate entity set and the number of groundtruth answers. Compared to BERT, WKLM is often less sensitive to these two factors.
9https://huggingface.co/pytorch-transformers

title: PRETRAINED ENCYCLOPEDIA: WEAKLY SUPERVISED KNOWLEDGE-PRETRAINED LANGUAGE MODEL


summaries: This paper tackles the problem of unsupervised object discovery, whereby a labeled dataset must be leveraged in order to then cluster an unlabeled dataset with a set of unknown categories. The paper contributes three main ideas to succeed at this task, namely 1) use of self-supervised learning to initialize the representations in a way that doesn't bias them to the labeled data, 2) a robust rank-based metric to generate estimates of similarity/dissimilarity along with consistency-based regularization to improve optimization, and 3) Joint optimization/refinement using a combination of labeled/unlabeled losses, as well as ability to learn incrementally without forgeting the original labeled classes. Results are shown on a range of datasets including OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN. The results demonstrate improvement over the current state of art for this task.

summaries: This paper addresses the problem of clustering unseen classes. To learn a robust feature extractor, this paper proposes a multi-stage training framework, which leverages different supervised manners in each stage. Specifically, they initialize the network using the self-supervised learning on the union of all available data and then further finetune it using labelled data. Based on this, they propose the rank statistics which leverages the activation knowledge on labelled classes and rank the activated dimensions. Unseen data having similar rank results are clustered to obtain the initial pseudo labels. Finally, the network is jointly optimized with the ground-truth and generated pseudo labels (the pseudo ones will be updated during training). Extensive experiments on 5 datasets show that their method has significant advantages over SOTA owing to the learned robust feature extractor.

summaries: The authors propose a methodology to discover new categories in an unlabeled dataset with the help of a label one. The authors propose the following methodology. First bootstrap some features using self-supervised learning on labeled and unlabeled data. Then transferring the knowledge of the labeled data to the unlabeled one by supposing that the representations of both are similar, the similarity being a rank statistic. Then using this knowledge a joint supervised-unsupervised objective.

abstractText: We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labeled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use rank statistics to transfer the model’s knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.

authors:
Charu C. Aggarwal
Chandan K. Reddy

title: Data Clustering: Algorithms and Applications

venue: CRC Press,

authors:
Rahaf Aljundi
Francesca Babiloni
Mohamed Elhoseiny
Marcus Rohrbach
Tinne Tuytelaars

title: Memory aware synapses: Learning what (not) to forget

authors:
David Arthur
Sergei Vassilvitskii

title: k-means++: the advantages of careful seeding

venue: In ACM-SIAM symposium on Discrete algorithms,

authors:
Jianlong Chang
Lingfeng Wang
Gaofeng Meng
Shiming Xiang
Chunhong Pan

title: Deep adaptive image clustering

authors:
Olivier Chapelle
Bernhard Scholkopf
Alexander Zien

title: Semi-Supervised Learning

authors:
Dorin Comaniciu
Peter Meer

title: Mean shift: A robust approach toward feature space analysis

venue: IEEE TPAMI,

authors:
Thomas Dean
Mark A. Ruzon
Mark Segal
Jonathon Shlens
Sudheendra Vijayanarasimhan
Jay Yagnik

title: Fast, accurate detection of 100,000 object classes on a single machine

venue: In CVPR,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jua Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Kamran Ghasedi Dizaji
Amirhossein Herandi
Cheng Deng
Weidong Cai
Heng Huang

title: Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization

authors:
Yanwei Fu
Tao Xiang
Yu-Gang Jiang
Xiangyang Xue
Leonid Sigal
Shaogang Gong

title: Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content

venue: IEEE Signal Processing Magazine,

authors:
Spyros Gidaris
Praveer Singh
Nikos Komodakis

title: Unsupervised representation learning by predicting image rotations

authors:
Kai Han
Andrea Vedaldi
Andrew Zisserman

title: Learning to discover novel visual categories via deep transfer clustering

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Yen-Chang Hsu
Zhaoyang Lv
Zsolt Kira

title: Deep image category discovery using a transferred similarity function

venue: arxiv,

title: Learning to cluster in order to transfer across domains and tasks

authors:
Yen-Chang Hsu
Zhaoyang Lv
Joel Schlosser
Phillip Odom
Zsolt Kira

title: Multi-class classification without multi-class labels

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Alexander Kolesnikov
Xiaohua Zhai
Lucas Beyer

title: Revisiting self-supervised visual representation learning

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

venue: Technical report,

title: The hungarian method for the assignment problem

venue: Naval research logistics quarterly,

authors:
Samuli Laine
Timo Aila

title: Temporal ensembling for semi-supervised learning

authors:
Brenden M. Lake
Ruslan Salakhutdinov
Joshua B. Tenenbaum

title: Human-level concept learning through probabilistic program induction

authors:
David Lopez-Paz
Marc’Aurelio Ranzato

title: Gradient episodic memory for continual learning

venue: In NeurIPS,

title: Some methods for classification and analysis of multivariate observations

venue: In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y. Ng

title: Reading digits in natural images with unsupervised feature learning

venue: In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning,

authors:
Andrew Y. Ng
Michael I. Jordan
Yair Weiss

title: On spectral clustering: Analysis and an algorithm

authors:
Avital Oliver
Augustus Odena
Colin Raffel
Ekin D. Cubuk
Ian J. Goodfellow

title: Realistic evaluation of deep semi-supervised learning algorithms

venue: NeurIPS,

authors:
Sinno Jialin Pan
Qiang Yang

title: A survey on transfer learning

venue: IEEE Transactions on Knowledge and Data Engineering,

authors:
Rasmus
Mathias M. Berglund
Mikko Honkala
Harri Valpola
Tapani Raiko

title: Semi-supervised learning with ladder networks

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
Georg Sperl
Christoph H Lampert

title: icarl: Incremental classifier and representation learning

authors:
Sylvestre-Alvise Rebuffi
Sebastien Ehrhardt
Kai Han
Andrea Vedaldi
Andrew Zisserman

title: Semi-supervised learning with scarce annotations

authors:
Konstantin Shmelkov
Cordelia Schmid
Karteek Alahari

title: Incremental learning of object detectors without catastrophic forgetting

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

authors:
Ilya Sutskever
James Martens
George Dahl
Geoffrey Hinton

title: On the importance of initialization and momentum in deep learning

authors:
Chuanqi Tan
Fuchun Sun
Tao Kong
Wenchang Zhang
Chao Yang
Chunfang Liu

title: A survey on deep transfer learning

venue: In International Conference on Artificial Neural Networks,

authors:
Antti Tarvainen
Harri Valpola

title: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

authors:
Laurens van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

authors:
Oriol Vinyals
Charles Blundell
Timothy Lillicrap
Koray Kavukcuoglu
Daan Wierstra

title: Matching networks for one shot learning

authors:
Karl Weiss
Taghi M. Khoshgoftaar
DingDing Wang

title: A survey of transfer learning

venue: Journal of Big Data,

authors:
Yongqin Xian
Christoph H. Lampert
Bernt Schiele
Zeynep Akata

title: Zero-shot learning - a comprehensive evaluation of the good, the bad and the ugly

authors:
Junyuan Xie
Ross Girshick
Ali Farhadi

title: Unsupervised deep embedding for clustering analysis

authors:
Jay Yagnik
Dennis Strelow
David A. Ross
Ruei sung Lin

title: The power of comparative reasoning

authors:
Bo Yang
Xiao Fu
Nicholas D. Sidiropoulos
Mingyi Hong

title: Towards k-means-friendly spaces: Simultaneous deep learning and clustering

authors:
Jianwei Yang
Devi Parikh
Dhruv Batra

title: Joint unsupervised learning of deep representations and image clusters

authors:
Xiaohua Zhai
Avital Oliver
Alexander Kolesnikov
Lucas Beyer

title: S4l: Self-supervised semisupervised learning

venue: For ImageNet,

heading: 1 INTRODUCTION

text:
Modern machine learning systems can match or surpass human-level performance in tasks such as image classification (Deng et al., 2009), but at the cost of collecting large quantities of annotated training data. Semi-supervised learning (SSL) (Oliver et al., 2018) can alleviate this issue by mixing labelled with unlabelled data, which is usually much cheaper to obtain. However, these methods still require some annotations for each of the classes that one wishes to learn. We argue this is not always possible in real applications. For instance, consider the task of recognizing products in supermarkets. Thousands of new products are introduced in stores every week, and it would be very expensive to annotate them all. However, new products do not differ drastically from the existing ones, so it should be possible to discover them automatically as they arise in the data. Unfortunately, machines are still unable to effectively learn new classes without manual annotations.
In this paper, we thus consider the problem of discovering new visual classes automatically, assuming that a certain number of classes are already known by the model (Hsu et al., 2018; 2019; Han et al., 2019). This knowledge comes in the form of a labelled dataset of images for a certain set of classes. Given that this data is labelled, off-the-shelf supervised learning techniques can be used to train a very effective classifier for the known classes, particularly if Convolutional Neural Networks (CNNs) are employed. However, this does not mean that the learned features are useful as a representation of the new classes. Furthermore, even if the representation transfers well, one still has the problem of identifying the new classes in an unlabelled dataset, which is a clustering problem.
We tackle these problems by introducing a novel approach that combines three key ideas (section 2 and fig. 1). The first idea is to pre-train the image representation (a CNN) using all available images, both labelled and unlabelled, using a self-supervised learning objective. Crucially, this objective does not leverage the known labels, resulting in features that are much less biased towards the labelled
∗indicates equal contribution
classes. Labels are used only after pre-training to learn a classifier specific to the labelled data as well as to fine-tune the deepest layers of the CNN, for which self-supervision is not as effective.
The second idea is a new approach to transfer the information contained in the labelled images to the problem of clustering the unlabelled ones. Information is transferred by sharing the same representation between labelled and unlabelled images, motivated by the fact that the new classes are often similar to the known ones. In more detail, pairs of unlabelled images are compared via their representation vectors. The comparison is done using robust rank statistics, by testing if two images share the same subset of k maximally activated representation components. This test is used to decide if two unlabelled images belong to the same (new) class or not, generating a set of noisy pairwise pseudo-labels. The pseudo-labels are then used to learn a similarity function for the unlabelled images.
The third and final idea is, after bootstrapping the representation, to optimise the model by minimizing a joint objective function, containing terms for both the labelled and unlabelled subsets, using respectively the given labels and the generated pseudo-labels, thus avoiding the forgetting issue that may arise with a sequential approach. A further boost is obtained by incorporating incremental learning of the discovered classes in the classification task, which allows information to flow between the labelled and unlabelled images.
We evaluate our method on several public benchmarks (section 3), outperforming by a large margin all existing techniques (section 4) that can be applied to this problem, demonstrating the effectiveness of our approach. We conclude the paper by summarizing our findings (section 5). Our code can be found at http://www.robots.ox.ac.uk/˜vgg/research/auto_novel.

heading: 2 METHOD

text:
Given an unlabelled dataset Du = {xui , i = 1, . . . ,M} of images xui ∈ R3×H×W , our goal is to automatically cluster the images into a number of classes Cu, which we assume to be known a priori. We also assume to have a second labelled image dataset Dl = {(xli, yli), i = 1, . . . , N} where yli ∈ {1, . . . , Cl} is the class label for image xli. We also assume that the set of Cl labelled classes is disjoint from the set of Cu unlabelled ones. While the statistics of Dl and Du thus differ, we hypothesize that a general notion of what constitutes a “good class” can be extracted from Dl and that the latter can be used to better cluster Du.
We approach the problem by learning an image representation Φ : x 7→ Φ(x) ∈ Rd in the form of a CNN. The goal of the representation is to help to recognize the known classes and to discover the new ones. In order to learn this representation, we combine three ideas, detailed in the next three sections.

heading: 2.1 SELF-SUPERVISED LEARNING

text:
Given that we have a certain number of labelled images Dl at our disposal, the obvious idea is to use these labels to bootstrap the representation Φ by minimizing a standard supervised objective such as the cross-entropy loss. However, experiments show that this causes the representation to overly-specialize for the classes in Dl, providing a poor representation of the new classes in Du.
Thus we resist the temptation of using the labels right away and use instead a self-supervised learning method to bootstrap the representation Φ. Self-supervised learning has been shown (Kolesnikov et al., 2019; Gidaris et al., 2018) to produce robust low-level features, especially for the first few layers of typical CNNs. It has the benefit that no data annotations are needed, and thus it can be applied to both labelled and unlabelled images during training. In this way, we achieve the key benefit of ensuring that the representation is initialized without being biased towards the labelled data.
In detail, we first pre-train our model Φ with self-supervision on the union of Dl and Du (ignoring all labels). We use the RotNet (Gidaris et al., 2018) approach1 due to its simplicity and efficacy, but any self-supervised method could be used instead. We then extend the pre-trained network Φ with a classification head ηl : Rd → RCl implemented as a single linear layer followed by a softmax layer. The function ηl ◦ Φ is fine-tuned on the labelled dataset Dl in order to learn a classifier for the Cl known classes, this time using the labels yi and optimizing the standard cross-entropy (CE) loss:
LCE = − 1
N N∑ i=1 log ηlyi(z l i) (1)
where zli = Φ(x l i) ∈ Rd is the representation of image xli. Only ηl and the last macro-block of Φ (section 3) are updated in order to avoid overfitting the representation to the labelled data.

heading: 2.2 TRANSFER LEARNING VIA RANK STATISTICS

text:
Once the representation Φ and the classifier ηl have been trained, we are ready to look for the new classes in Du. Since the classes in Du are unknown, we represent them by defining a relation among pairs of unlabelled images (xui , x u j ). The idea is that similar images should belong to the same (new) class, which we denote by the symbol sij = 1, while dissimilar ones should not, which we denote by sij = 0. The problem is then to obtain the labels sij .
Our assumption is that the new classes will have some degree of visual similarity with the known ones. Hence, the learned representation should be applicable to old and new classes equally well. As a consequence, we expect the descriptors zui = Φ(x u i ) and z u j = Φ(x u j ) of two images x u i , x u j from the new classes to be close if they are from the same (new) class, and to be distinct otherwise.
Rather than comparing vectors zui , z u j directly (e.g., by a scalar product), however, we use a more robust rank statistics. Specifically, we rank the values in vector zui by magnitude. Then, if the rankings obtained for two unlabelled images xui and x u j are the same, they are very likely to belong to the same (new) class, so we set sij = 1. Otherwise, we set sij = 0. In practice, it is too strict to require the two rankings to be identical if the dimension of zui is high (otherwise we may end up with sij = 0 for all pairs (i, j), i 6= j). Therefore, we relax this requirement by only testing if the sets of the top-k ranked dimensions are the same (we use k = 5 in our experiments), i.e.:
sij = 1 { topk(Φ(x u i )) = topk(Φ(x u j )) } , (2)
where topk : Rd → P({1, . . . , d}) associates to a vector z the subset of indices {1, . . . , d} of its top-k elements.
Once the labels sij have been obtained, we use them as pseudo-labels to train a comparison function for the unlabelled data. In order to do this, we apply a new head ηu : Rd → RCu to the image representation zui = Φ(x u i ) to extract a new descriptor vector η
u(zui ) optimized for the unlabelled data. As in section 2.1, the head is composed of a linear layer followed by a softmax. Then, the inner product ηu(zui ) >ηu(zuj ) is used as a score for whether images x u i and x u j belong to the same class or not. Note that ηu(zui ) is a normalized vector due to the softmax layer in η u. This descriptor is trained by optimizing the binary cross-entropy (BCE) loss:
LBCE = − 1
M2 M∑ i=1 M∑ j=1 [sij log η u(zui ) >ηu(zuj ) + (1− sij) log(1− ηu(zui )>ηu(zuj ))]. (3)
1 We present to the network Φ randomly-rotated versions Rx of each image and task it with predicting R. The problem is formulated as a 4-way classification of the rotation angle, with angle in {0◦, 90◦, 180◦, 270◦}. The model η ◦ Φ(Rx) is terminated by a single linear layer η with 4 outputs each scoring an hypothesis. The parameters of η and Φ are optimized by minimizing the cross-entropy loss on the rotation prediction.
Furthermore, we structure ηu in a particular manner: We set its output dimension to be equal to the number of new classes Cu. In this manner, we can use the index of the maximum element of each vector ŷui = argmaxy[η
u ◦ Φ(xui )]y as prediction ŷui for the class of image xui (as opposed to assigning labels via a clustering method such as k-means).

heading: 2.3 JOINT TRAINING ON LABELLED AND UNLABELLED DATA

text:
We now have two losses that involve the representation Φ: the CE loss LCE for the labelled data Dl and the pairwise BCE loss LBCE for the unlabelled data Du. They both share the same image embedding Φ. This embedding can be trained sequentially, first on the labelled data, and then on the unlabelled data using the pseudo-labels obtained above. However, in this way the model will very likely forget the knowledge learned from the labelled data, which is known as catastrophic forgetting in incremental learning (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Shmelkov et al., 2017; Aljundi et al., 2018).
In contrast, we jointly fine-tune our model using both losses at the same time. Note that most of the model Φ is frozen; we only fine-tune the last macro-block of Φ together with the two heads ηu and ηl. Importantly, as we fine-tune the model, the labels sij are changing at every epoch as the embedding ηl is updated. This in turn affects the rank statistics used to determine the labels sij as explained in section 2.2. This leads to a “moving target” phenomenon that can introduce some instability in learning the model. This potential issue is addressed in the next section.

heading: 2.4 ENFORCING PREDICTIONS TO BE CONSISTENT

text:
In addition to the CE and BCE losses, we also introduce a consistency regularization term, which is used for both labelled and unlabelled data. In semi-supervised learning (Oliver et al., 2018; Tarvainen & Valpola, 2017; Laine & Aila, 2017), the idea of consistency is that the class predictions on an image x and on a randomly-transformed counterpart tx (for example an image rotation) should be the same. In our case, as will be shown in the experiments, consistency is very important to obtain good performance. One reason is that, as noted above, the pairwise pseudo-labels for the unlabelled data are subject to change on the fly during training. Indeed, for an image xui and a randomly-transformed counterpart txui , if we do not enforce consistency, we can have topk(Φ(x u i )) 6= topk(Φ(txui )). According to eq. (2) defining sij , it could result in different sij for xui depending on the data augmentation applied to the image. This variability of the ranking labels for a given pair could then confuse the training of the embedding.
Following the common practice in semi-supervised learning, we use the Mean Squared Error (MSE) as the consistency cost. This is given by:
LMSE = 1
N N∑ i=1 (ηl(zli)− ηl(ẑli))2 + 1 M M∑ i=1 (ηu(zui )− ηu(ẑui ))2, (4)
where ẑ is the representation of tx.
The overall loss of our model can then be written as L = LCE + LBCE + ω(t)LMSE, (5)
where the coefficient ω(t) is a ramp-up function. This is widely used in semi-supervised learning (Laine & Aila, 2017; Tarvainen & Valpola, 2017). Following (Laine & Aila, 2017; Tarvainen & Valpola, 2017), we use the sigmoid-shaped function ω(t) = λe−5(1− t T ) 2
, where t is current time step and T is the ramp-up length and λ ∈ R+.

heading: 2.5 INCREMENTAL LEARNING SCHEME

text:
We also explore a setting analogous to incremental learning. In this approach, after tuning on the labelled set (end of section 2.1), we extend the head ηl to Cu new classes, so that ηl : Rd → RCl+Cu . The head parameters for the new classes are initialized randomly. The model is then trained using the same loss eq. (5), but the cross-entropy part of the loss is evaluated on both labelled and unlabelled data Dl and Du. Since the cross-entropy requires labels, for the unlabelled data we use the pseudo-labels ŷui , which are generated on-the-fly from the head η u at each forward pass.
The advantage is that this approach increments ηl to discriminate both old and new classes, which is often desirable in applications. It also creates a feedback loop that causes the features zui to be
CIFAR-10 CIFAR-100 SVHN
Ours w/o Con 82.6±12.0% 61.8±3.6% 61.3±1.9% Ours w/o CE 84.7±4.4% 58.4±2.7% 59.7±6.6% Ours w/o BCE 26.2±2.0% 6.6±0.7% 24.5±0.5% Ours w/o S.S. 89.4±1.4% 67.4±2.0% 72.9±5.0% Ours full 90.4±0.5% 73.2±2.1% 95.0±0.2% Ours w/ I.L. 91.7±0.9% 75.2±4.2% 95.2±0.3%
Table 1: Ablation study. “w/o Con.” means without consistency constraints; “w/o CE” means without the cross entropy loss for training on labeled data. “w/o BCE” means without binary cross entropy loss for training on unlabeled data; “w/o S.S.” means without selfsupervision.
refined, which in turn generates better pseudo-labels ŷui for D u from the head ηu. In this manner, further improvements can be obtained by this cycle of positive interactions between the two heads during training.

heading: 3 EXPERIMENTS

heading: 3.1 DATA AND EXPERIMENTAL DETAILS

text:
We evaluate our models on a variety of standard benchmark datasets: CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), OmniGlot (Lake et al., 2015), and ImageNet (Deng et al., 2009). Following Han et al. (2019), we split these to have 5/20/5/654/30 classes respectively in the unlabelled set. In addition, for OmniGlot and ImageNet we use 20 and 3 different splits respectively, as in Han et al. (2019), and report average clustering accuracy. More details on the splits can be found in appendix A.
Evaluation metrics. We adopt clustering accuracy (ACC) to evaluate the clustering performance of our approach. The ACC is defined as follow:
max g∈Sym(L)
1
N N∑ i=1 1 {yi = g (yi)} , (6)
where yi and yi denote the ground-truth label and clustering assignment for each data point x u i ∈ Du respectively, and Sym(L) is the group permutations of L elements (this discounts the fact that the cluster indices may not be in the same order as the ground-truth labels). Permutations are optimized using the Hungarian algorithm (Kuhn, 1955).
Implementation details. We use the ResNet-18 (He et al., 2016) architecture, except for OmniGlot for which we use a VGG-like network (Simonyan & Zisserman, 2015) with six layers to make our setting directly comparable to prior work. We use SGD with momentum (Sutskever et al., 2013) as optimizer for all but the OmniGlot dataset, for which we use Adam (Kingma & Ba, 2014). For all experiments we use a batch size of 128 and k = 5 which we found worked consistently well across datasets (see appendix D). More details about the hyper-parameters can be found in appendix B.

heading: 3.2 ABLATION STUDY

text:
We validate the effectiveness of the components of our method by ablating them and measuring the resulting ACC on the unlabelled data. Note that, since the evaluation is restricted to the unlabelled data, we are solving a clustering problem. The same unlabelled data points are used for both training and testing, except that data augmentation (i.e. image transformations) is not applied when computing the cluster assignments. As can be seen in table 1, all components have a significant effect as removing any of them causes the performance to drop substantially. Among them, the BCE loss is by far the most important one, since removing it results in a dramatic drop of 40–60% absolute ACC points. For example, the full method has ACC 90.4% on CIFAR-10, while removing BCE causes the ACC to drop to 26.2%. This shows that that our rank-based embedding comparison can indeed generate reliable pairwise pseudo labels for the BCE loss. Without consistency, cross entropy, or self-supervision, the performance drops by a more modest but still significant 7.8%, 5.7% and 1.0% absolute ACC points, respectively, for CIFAR-10. It means that the consistency term plays a role as important as the cross-entropy term by preventing the “moving target” phenomenon described in section 2.4. Finally, by incorporating the discovered classes in the classification task, we get a further boost of 1.3%, 2.0% and 0.2% points on CIFAR-10, CIFAR-100 and SVHN respectively.
Table 2: Novel category discovery results on CIFAR-10, CIFAR-100, and SVHN. ACC on the unlabelled set. “w/ S.S.” means with self-supervised learning.
No CIFAR-10 CIFAR-100 SVHN
(1) k-means (MacQueen, 1967) 65.5±0.0 % 56.6±1.6% 42.6%±0.0 (2) KCL (Hsu et al., 2018) 66.5±3.9% 14.3±1.3% 21.4%±0.6 (3) MCL (Hsu et al., 2019) 64.2±0.1% 21.3±3.4% 38.6%±10.8 (4) DTC (Han et al., 2019) 87.5±0.3% 56.7±1.2% 60.9%±1.6 (5) k-means (MacQueen, 1967) w/ S.S. 72.5±0.0% 56.3±1.7% 46.7±0.0% (6) KCL (Hsu et al., 2018) w/ S.S. 72.3±0.2% 42.1±1.8% 65.6±4.9% (7) MCL (Hsu et al., 2019) w/ S.S. 70.9±0.1% 21.5±2.3% 53.1±0.3% (8) DTC (Han et al., 2019) w/ S.S. 88.7±0.3% 67.3±1.2% 75.7±0.4% (9) Ours 90.4±0.5% 73.2±2.1% 95.0±0.2% (10) Ours w/ I.L. 91.7±0.9% 75.2±4.2% 95.2±0.2%
(a) init (b) epoch 30 (c) epoch 90
Figure 2: Evolution of the t-SNE during the training of CIFAR-10. Performed on unlabelled data (i.e., instances of dog, frog, horse, ship, truck). Colors of data points denote their ground-truth labels.

heading: 3.3 NOVEL CATEGORY DISCOVERY

text:
We compare our method to baselines and state-of-the-art methods for new class discovery, starting from CIFAR-10, CIFAR-100, and SVHN in table 2. The first baseline (row 5 in table 2) amounts to applying k-means (MacQueen, 1967) to the features extracted by the fine-tuned model (the second step in section 2.1), for which we use the k-means++ (Arthur & Vassilvitskii, 2007) initialization. The second baseline (row 1 in table 2) is similar, but uses as feature extractor a model trained from scratch using only the labelled images, which corresponds to a standard transfer learning setting. By comparing rows 1, 5 and 9 in table 2, we can see that our method substantially outperforms k-means. Next, we compare with the KCL (Hsu et al., 2018), MCL (Hsu et al., 2019) and DTC (Han et al., 2019) methods. By comparing rows 2–4 to 9, we see that our method outperforms these by a large margin. We also try to improve KCL, MCL and DTC by using the same self-supervised initialization we adopt (section 2.1), which indeed results in an improvement (rows 2–4 vs 6–8). However, their overall performance still lags behind ours by a large margin. For example, our method of section 2.4 achieves 95.0% ACC on SVHN, while “KCL w/ S.S.”, “MCL w/ S.S.” and “DTC w/ S.S.” achieve only 65.6%, 53.1% and 75.7% ACC, respectively. Similar trends hold for CIFAR-10 and CIFAR-100. Finally, the incremental learning scheme of section 2.5 results in further improvements, as can be seen by comparing rows 9 and 10 of table 2.
In fig. 2, we show the evolution of the learned representation on the unlabelled data on CIFAR-10 using t-SNE (van der Maaten & Hinton, 2008). As can be seen, while the clusters overlap in the beginning, they become more and more separated as the training progresses, showing that our model can effectively discover novel visual categories without labels and learn meaningful embeddings for them.
We further compare our method to others on two more challenging datasets, OmniGlot and ImageNet, in table 3. For OmniGlot, results are averaged over the 20 alphabets in the evaluation set (see appendix A); for ImageNet, results are averaged over the three 30-class unlabelled sets used in (Hsu et al., 2018; 2019). Since we have a relatively larger number of labelled classes in these two datasets, we follow (Han et al., 2019) and use metric learning on the labelled classes to pre-train the feature extractor, instead of the self-supervised learning. We empirically found that self-supervision does not provide obvious gains for these two datasets. This is reasonable since the data in the labelled sets of these two datasets are rather diverse and abundant, so metric learning can provide good feature initialization as there is less class-specific bias due to the large number of pre-training classes. However, by comparing rows 1 and 5 in table 3, it is clear that metric learning alone is not sufficient for the task of novel category discovery. Our method substantially outperforms the k-means results obtained using the features from metric learning — by 11.9% and 10.6% on OmniGlot and ImageNet respectively. Our method also substantially outperforms the current state-of-the-art, achieving 89.1%
No OmniGlot ImageNet
(1) k-means (MacQueen, 1967) 77.2% 71.9% (2) KCL (Hsu et al., 2018) 82.4% 73.8% (3) MCL (Hsu et al., 2019) 83.3% 74.4% (4) DTC (Han et al., 2019) 89.0% 78.3%
(5) Ours 89.1% 82.5%
Table 3: Novel category discovery results on OmniGlot and ImageNet. ACC on the unlabelled set.
(a) Ours (b) + incr. learning
Figure 3: t-SNE on CIFAR-10: impact of incremental Learning. (a) representation on the labelled and (b) unlabelled CIFAR classes. Colors of data points denote their ground-truth labels. We observe a bigger overlap in (a) between the “old” class 3 and the “new” class 5 when not incorporating Incremental Learning.
and 82.5% ACC on OmniGlot and ImageNet respectively, compared with 89.0% and 78.8% of (Han et al., 2019), thus setting the new state-of-the-art.

heading: 3.4 INCREMENTAL LEARNING

text:
Here, we further evaluate our incremental scheme for novel category discovery as described in section 2.5. Methods for novel category discovery such as (Han et al., 2019; Hsu et al., 2019; 2018) focus on obtaining the highest clustering accuracy for the new unlabelled classes, but may forget the existing labelled classes in the process. In practice, forgetting is not desirable as the model should be able to recognize both old and new classes. Thus, we argue that the classification accuracy on the labelled classes should be assessed as well, as for any incremental learning setting. Note however that our setup differs substantially from standard incremental learning (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Shmelkov et al., 2017; Aljundi et al., 2018) where every class is labelled and the focus is on using limited memory. In our case, we can store and access the original data without memory constraints, but the new classes are unlabelled, which is often encountered in applications.
By construction (section 2.5), our method learns the new classes on top of the old ones incrementally, out of the box. In order to compare to methods such as KCL, MCL and DTC that do not have this property, we proceed as follows. First, the method runs as usual to cluster the unlabelled portion of the data, thus obtaining pseudo-labels for it, and learning a feature extractor as a byproduct. Then, the feature extractor is used to compute features for both the labelled and unlabelled training data, and a linear classifier is trained using labels and pseudo-labels, jointly on all the classes, old and new.
We report in table 4 the performance of the resulting joint classifier networks on the testing set of each dataset (this is now entirely disjoint from the training set). Our method has similar performances on the old and new classes for CIFAR-10 and SVHN, as might be expected as the split between old and new classes is balanced. In comparison, the feature extractor learned by KCL and MCL works much better for the old classes (e.g., the accuracy discrepancy between old and new classes is 25.3% for KCL on SVHN). Conversely, DTC learns features that work better for the new classes, as shown by the poor performance for the old classes on CIFAR-10. Thus, KCL, MCL and DTC learn representations that are biased to either the old or new classes, resulting overall in suboptimal performance. In contrast, our method works well on both old and new classes; furthermore, it drastically outperforms existing methods on both.

heading: 4 RELATED WORK

text:
Our work draws inspiration from semi-supervised learning, transfer learning, clustering, and zero-shot learning. We review below the most relevant contributions.
In semi-supervised learning (SSL) (Chapelle et al., 2006), a partially labelled training dataset is given and the objective is to learn a model that can propagate the labels from the labelled data to unlabelled data. Most SSL methods focus on the classification task where, usually both labelled and unlabelled points belong to the same set of classes. On the contrary, our goal is to handle the case where the unlabelled data classes differ from the labelled data. Oliver et al. (2018) summarizes the state-of-the-art SSL methods. Among them, the consistency-based methods appeared to be the most effective. Rasmus et al. (2015) propose a ladder network which is trained on both labelled and unlabelled data using a reconstruction loss. Laine & Aila (2017) simplifies this ladder network by enforcing prediction consistency between a data point and its augmented counterpart. As an alternative to data augmentation, they also consider a regularization method based on the exponential moving average (EMA) of the predictions. This idea is further improved by Tarvainen & Valpola (2017): instead of using the EMA of predictions, they propose to maintain the EMA of model parameters. The consistency is then measured between the predictions of the current model (student) and the predictions of the EMA model (teacher). More recently (and closer to our work) practitioners have also combined SSL with self-supervision(Rebuffi et al., 2019; Zhai et al., 2019) to leverage dataset with very few annotations.
Transfer learning (Pan & Yang, 2010; Weiss et al., 2016; Tan et al., 2018) is an effective way to reduce the amount of data annotations required to train a model by pre-training the model on an existing dataset. In image classification, for example, it is customary to start from a model pre-trained on the ImageNet (Deng et al., 2009) dataset. In most transfer learning settings, however, both the source data and the target data are fully annotated. In contrast, our goal is to transfer information from a labelled dataset to an unlabelled one.
Many classic (e.g., Aggarwal & Reddy (2013); MacQueen (1967); Comaniciu & Meer (1979); Ng et al. (2001)) and deep learning (e.g., Xie et al. (2016); Chang et al. (2017); Dizaji et al. (2017); Yang et al. (2017; 2016); Hsu et al. (2018; 2019)) clustering methods have been proposed to automatically partition an unlabelled data collection into different classes. However, this task is usually ill-posed as there are multiple, equally valid criteria to partition most datasets. We address this challenge by learning the appropriate criterion by using a labelled dataset, narrowing down what constitutes a proper class. We call this setting “transfer clustering”.
To the best of our knowledge, the work most related to ours are (Hsu et al., 2018; 2019; Han et al., 2019). Han et al. (2019) also consider discovering new classes as a transfer clustering problem. They first learn a data embedding by using metric learning on the labelled data, and then fine-tune the embedding and learn the cluster assignments on the unlabelled data. In (Hsu et al., 2018; 2019), the authors introduce KCL and MCL clustering methods. In both, a similarity prediction network (SPN), also used in (Hsu et al., 2016), is first trained on a labelled dataset. Afterwards, the pre-trained SPN is used to provide binary pseudo labels for training the main model on an unlabelled dataset. The overall pipelines of the two methods are similar, but the losses differ: KCL uses a Kullback-Leibler divergence based contrastive loss equivalent to the BCE used in this paper (eq. (3)), and MCL uses the Meta Classification Likelihood loss. Zero-shot learning (ZSL) (Xian et al., 2018; Fu et al., 2018) can also be used to recognize new classes. However, differently from our work, ZSL also requires additional side information (e.g., class attributes) in addition to the raw images.
Finally, other works (Dean et al., 2013; Yagnik et al., 2011) discuss the application of rank statistics to measuring the similarity of vectors; however, to the best of our knowledge, we are the first to apply rank statistics to the task of novel category discovery using deep neural networks.

heading: 5 CONCLUSIONS

text:
In this paper, we have looked at the problem of discovering new classes in an image collection, leveraging labels available for other, known classes. We have shown that this task can be addressed very successfully by a few new ideas. First, the use of self-supervised learning for bootstrapping the image representation trades off the representation quality with its generality, and for our problem this leads to a better solution overall. Second, we have shown that rank statistics are an effective method to compare noisy image descriptors, resulting in robust data clustering. Third, we have
shown that jointly optimizing both labelled recognition and unlabelled clustering in an incremental learning setup can reinforce the two tasks while avoiding forgetting. On standard benchmarks, the combination of these ideas results in much better performance than existing methods that solve the same task. Finally, for larger datasets with more classes and diverse data (e.g., ImageNet) we note that self-supervision can be bypassed as the pretraining on labelled data already provides a powerful enough representation. In such cases, we still show that the rank statistics for clustering gives drastic improvement over existing methods.

heading: 6 ACKNOWLEDGMENTS

text: This work is supported by the EPSRC Programme Grant Seebibyte EP/M013774/1, Mathworks/DTA DFR02620, and ERC IDIU-638009.

heading: A DATASET SPLITS

text:
For CIFAR-10 and SVHN we keep the labels of the five first categories (namely airplane, automobile, bird, cat, deer for CIFAR-10, 0–4 for SVHN) and keep the rest of the data as the unlabelled set. For CIFAR-100 we use the first 80 categories as labelled data while the rest are used for the unlabelled set. Following Hsu et al. (2018; 2019), for OmniGlot, each of the 20 alphabets in evaluation set (with 20–47 categories, 659 characters/class) is used as unlabelled data, and all the 30 alphabets in background set are used as labelled set (964 characters/class). For ImageNet, we follow Hsu et al. (2018; 2019) to use the 882/118 classes split proposed in Vinyals et al. (2016), and use the three 30-class subsets sampled from the 118 classes as unlabelled sets.
B IMPLEMENTATION DETAILS In the first self-supervised training step, otherwise mentioned, we trained our model with the pretext task of rotation predictions (i.e., a four-class classification: 0◦, 90◦, 180◦, and 270◦) for 200 epochs and a step-wise decaying learning rate starting from 0.1 and divided by 5 at epochs 60, 120, and 160.
In the second step of our framework (i.e., supervised training using labelled data), we fine-tuned our model on the labelled set for 100 epochs and a step-wise decaying learning rate starting from 0.1 and halved every 10 epochs. From this step onward we fix the first three convolutional blocks of the model, and fine-tuned the last convolutional block together with the linear classifier.
Finally, in the last joint training step, we fine-tuned our model for 200/100/90 epochs for {CIFAR-10, CIFAR-100, SVHN}/OmniGlot/ImageNet, which was randomly sampled from the merged set of both labelled and unlabelled data. The initial learning rate was set to 0.1 for all datasets, and was decayed with a factor of 10 at the 170th/{30th, 60th} epoch for {CIFAR-10, CIFAR-100, SVHN}/ImageNet. The learning rate of 0.01 was kept fixed for OmniGlot. For the consistency regularization term, we used the ramp-up function as described in section 2.4 with λ = {5.0, 50.0, 50.0, 100.0, 10.0}, and T = {50, 150, 80, 1, 50} for CIFAR-10, CIFAR-100, SVHN, OmniGlot, and ImageNet respectively. In the incremental learning setting, all previous hyper parameters remain the same for our method. We only add a ramp-up on the cross entropy loss on unlabelled data. The ramp-up length is the same as the one used for eq. (4) and we use for all experiments a coefficient of 0.05. For all other methods we trained the classifier for 150 epochs with SGD with momentum and learning rate of 0.1 divided by 10 at epoch 50.
We implemented our method using PyTorch 1.1.0 and ran experiments on NVIDIA Tesla M40 GPUs. Following (Han et al., 2019), our results were averaged over 10 runs for all datasets, except for ImageNet which was averaged over the three 30-class subsets. In general, we found the results were stable. Our code is publicly available at http://www.robots.ox.ac.uk/˜vgg/ research/auto_novel.

heading: C STANDARD DEVIATION OF INCREMENTAL LEARNING EXPERIMENT

text:
IN TABLE 4
D IMPACT OF k OVER RESULTS
We provide an additional study of the evolution of performances of our method with respect to k. We results on SVHN/CIFAR10/CIFAR100 in fig. 4. We found that k = {5, 7} gave the best results overall. We also found that for all values of k except 1 results were in general stable.

heading: E RESULTS WITH AN UNKNOWN NUMBER OF CLASSES

text: While in our work we assume the number of new classes Cu to be known a priori, this hypothesis can be restrictive in practice. Instead, one can estimate the number of classes in Du using recent methods such as DTC (Han et al., 2019). In table 6 we compare ACC of KCL (Hsu et al., 2018), KCL (Hsu et al., 2019), DTC(Han et al., 2019) and our method on unlabelled splits of OmniGlot and ImageNet datasets with Cu computed from DTC. We note that our method again reaches the state-of-the-art on ImageNet and is on par with the state-of-the-art on OmniGlot.