  It is well known that data scientists spend the majority of their time on
preparing data for analysis. One of the first steps in this preparation phase
is to load the data from the raw storage format. Comma-separated value (CSV)
files are a popular format for tabular data due to their simplicity and
ostensible ease of use. However, formatting standards for CSV files are not
followed consistently, so each file requires manual inspection and potentially
repair before the data can be loaded, an enormous waste of human effort for a
task that should be one of the simplest parts of data science. The first and
most essential step in retrieving data from CSV files is deciding on the
dialect of the file, such as the cell delimiter and quote character. Existing
dialect detection approaches are few and non-robust. In this paper, we propose
a dialect detection method based on a novel measure of data consistency of
parsed data files. Our method achieves 97% overall accuracy on a large corpus
of real-world CSV files and improves the accuracy on messy CSV files by almost
22% compared to existing approaches, including those in the Python standard
library.

Databases