  We present the information-ordered bottleneck (IOB), a neural layer designed
to adaptively compress data into latent variables ordered by likelihood
maximization. Without retraining, IOB nodes can be truncated at any bottleneck
width, capturing the most crucial information in the first latent variables.
Unifying several previous approaches, we show that IOBs achieve near-optimal
compression for a given encoding architecture and can assign ordering to latent
signals in a manner that is semantically meaningful. IOBs demonstrate a
remarkable ability to compress embeddings of image and text data, leveraging
the performance of SOTA architectures such as CNNs, transformers, and diffusion
models. Moreover, we introduce a novel theory for estimating global intrinsic
dimensionality with IOBs and show that they recover SOTA dimensionality
estimates for complex synthetic data. Furthermore, we showcase the utility of
these models for exploratory analysis through applications on heterogeneous
datasets, enabling computer-aided discovery of dataset complexity.

Machine Learning