Institutional Books 1.0: A 242B token dataset from Harvard Library's
  collections, refined for accuracy and usability
Large language models (LLMs) use data to learn about the world in order to
produce meaningful correlations and predictions. As such, the nature, scale,
quality, and diversity of the datasets used to train these models, or to
support their work at inference time, have a direct impact on their quality.
The rapid development and adoption of LLMs of varying quality has brought into
focus the scarcity of publicly available, high-quality training data and
revealed an urgent need to ground the stewardship of these datasets in
sustainable practices with clear provenance chains. To that end, this technical
report introduces Institutional Books 1.0, a large collection of public domain
books originally digitized through Harvard Library's participation in the
Google Books project, beginning in 2006. Working with Harvard Library, we
extracted, analyzed, and processed these volumes into an extensively-documented
dataset of historic texts. This analysis covers the entirety of Harvard
Library's collection scanned as part of that project, originally spanning
1,075,899 volumes written in over 250 different languages for a total of
approximately 250 billion tokens. As part of this initial release, the
OCR-extracted text (original and post-processed) as well as the metadata
(bibliographic, source, and generated) of the 983,004 volumes, or 242B tokens,
identified as being in the public domain have been made available. This report
describes this project's goals and methods as well as the results of the
analyses we performed, all in service of making this historical collection more
accessible and easier for humans and machines alike to filter, read and use.
684954493614057188acbf5a
Matteo Cargnelutti
claimed_verified
2025-06-13T07:41:51.757Z
64dbb1eba8829bc784f8b5d5
https://cdn-avatars.huggingface.co/v1/production/uploads/64dbb1eba8829bc784f8b5d5/kitQ3XjvztfcvxIQJQ958.jpeg
Matteo Cargnelutti
user
MatteoCargnelutti
684954493614057188acbf5b
Catherine Brobston
684954493614057188acbf5c
John Hess
684954493614057188acbf5d
Jack Cushman
684954493614057188acbf5e
Kristi Mukk
684954493614057188acbf5f
Aristana Scourtas
684954493614057188acbf60
Kyle Courtney
684954493614057188acbf61
Greg Leppert
684954493614057188acbf62
Amanda Watson
684954493614057188acbf63
Martha Whitehead
684954493614057188acbf64
Jonathan Zittrain
Institutional Books 1.0: A 242B token dataset from Harvard Library's
  collections, refined for accuracy and usability
Large language models (LLMs) use data to learn about the world in order to
produce meaningful correlations and predictions. As such, the nature, scale,
quality, and diversity of the datasets used to train these models, or to
support their work at inference time, have a direct impact on their quality.
The rapid development and adoption of LLMs of varying quality has brought into
focus the scarcity of publicly available, high-quality training data and
revealed an urgent need to ground the stewardship of these datasets in
sustainable practices with clear provenance chains. To that end, this technical
report introduces Institutional Books 1.0, a large collection of public domain
books originally digitized through Harvard Library's participation in the
Google Books project, beginning in 2006. Working with Harvard Library, we
extracted, analyzed, and processed these volumes into an extensively-documented
dataset of historic texts. This analysis covers the entirety of Harvard
Library's collection scanned as part of that project, originally spanning
1,075,899 volumes written in over 250 different languages for a total of
approximately 250 billion tokens. As part of this initial release, the
OCR-extracted text (original and post-processed) as well as the metadata
(bibliographic, source, and generated) of the 983,004 volumes, or 242B tokens,
identified as being in the public domain have been made available. This report
describes this project's goals and methods as well as the results of the
analyses we performed, all in service of making this historical collection more
accessible and easier for humans and machines alike to filter, read and use.
Institutional Books 1.0 is a large dataset of public domain books, totaling approximately 242 billion tokens, made available through OCR and metadata processing to facilitate accessibility and usage.
Large language models Institutional Books 1.0 Harvard Library Google Books project OCR-extracted text metadata bibliographic source generated public domain