Robert_Crowe:


Do the pixels have a minimum level of variance (for example, did we take the lens cap off?)



Could you expand a bit more on this? I agree with the rest of the points mentioned for validation. Checking for format corruption might be another good addition.
Anomaly detection for vision datasets is way more nuanced than many other modalities I believe. For text-based problems, it’s easier to compute the descriptive statistics and compare examples against those but just getting the mean and std of a couple million images can be an expensive operation (given we operating on high-res images like 224x224x3).
Another post-training technique to discard some of the anomalies inside the training set is to plot the samples that cause a model to incur high loss values. Here’s 2 a PoC of this technique. I learned this at one of the fast.ai classes.
Computing the distribution overlap is also not very straightforward for large vision datasets. In my experience, simple recipes like computing histogram frequencies often fail to capture the important trends.
One cheap but often effective technique I have seen on Kaggle (for tabular datasets in most of the cases) is to train a simple model to classify whether a given data point is from the training distribution. How this works in practice? We take the entries of the training set, discard their labels, and indicate 1s as their new labels to all the training data points. This is the training data for the simple model now. We take the trained model to infer on the validation and test sets to investigate the overlap.
All of this probably calls for a well-structured paper 
For feature engineering, I think for most DL-related vision workflows users would want to have their own data augmentation pipelines incorporated.