title: Challenges with unsupervised LLM knowledge discovery
summary: We show that existing unsupervised methods on large language model (LLM)
  activations do not discover knowledge -- instead they seem to discover whatever
  feature of the activations is most prominent. The idea behind unsupervised
  knowledge elicitation is that knowledge satisfies a consistency structure,
  which can be used to discover knowledge. We first prove theoretically that
  arbitrary features (not just knowledge) satisfy the consistency structure of a
  particular leading unsupervised knowledge-elicitation method,
  contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a
  series of experiments showing settings in which unsupervised methods result in
  classifiers that do not predict knowledge, but instead predict a different
  prominent feature. We conclude that existing unsupervised methods for
  discovering latent knowledge are insufficient, and we contribute sanity checks
  to apply to evaluating future knowledge elicitation methods. Conceptually, we
  hypothesise that the identification issues explored here, e.g. distinguishing a
  model's knowledge from that of a simulated character's, will persist for future
  unsupervised methods.
paper_authors._id: 657fb5e79a9e347d4bf6378d
paper_authors.name: Sebastian Farquhar
paper_authors._id: 657fb5e79a9e347d4bf6378e
paper_authors.name: Vikrant Varma
paper_authors._id: 657fb5e79a9e347d4bf6378f
paper_authors.name: Zachary Kenton
paper_authors._id: 657fb5e79a9e347d4bf63790
paper_authors.name: Johannes Gasteiger
paper_authors._id: 657fb5e79a9e347d4bf63791
paper_authors.name: Vladimir Mikulik
paper_authors._id: 657fb5e79a9e347d4bf63792
paper_authors.name: Rohin Shah
paper_title: Challenges with unsupervised LLM knowledge discovery
paper_summary: We show that existing unsupervised methods on large language model (LLM)
  activations do not discover knowledge -- instead they seem to discover whatever
  feature of the activations is most prominent. The idea behind unsupervised
  knowledge elicitation is that knowledge satisfies a consistency structure,
  which can be used to discover knowledge. We first prove theoretically that
  arbitrary features (not just knowledge) satisfy the consistency structure of a
  particular leading unsupervised knowledge-elicitation method,
  contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a
  series of experiments showing settings in which unsupervised methods result in
  classifiers that do not predict knowledge, but instead predict a different
  prominent feature. We conclude that existing unsupervised methods for
  discovering latent knowledge are insufficient, and we contribute sanity checks
  to apply to evaluating future knowledge elicitation methods. Conceptually, we
  hypothesise that the identification issues explored here, e.g. distinguishing a
  model's knowledge from that of a simulated character's, will persist for future
  unsupervised methods.
paper_ai_keywords: unsupervised methods large language model (LLM) activations knowledge elicitation consistency structure contrast-consistent search classifiers latent knowledge sanity checks