Source needed: Why vector representation are ideal for statistical models I am just looking for a scientific source, either a paper or book, that I can refer to when talking about the fact that the ability to represent data in vectors is a desired property.
Reasons can include that most statistical learning algorithms, outside of representation learning, rely on linear algebra. Further, software and hardware are optimized toward matrix multiplications.

This is perhaps difficult, if not outright impossible, to pinpoint any particular source that exactly caters to OP's requirements.
However, $[\rm I]$ compiles the chronological infusion of matrix algebra and records certain quotes that air the necessity of the same (emphasis mine):

Bartlett $(1947)$ makes considerable use of matrices in his
long "Multivariate Statistics" paper wherein he writes that he has "avoided complicated
analytical discussion of theory" but has made use of "matrix and vector algebra".

This implicitly hinges on the power of the matrix algebra in place of scalar summation treatments:

It is
in Snedecor and Cochran ($1989, ~8$th Ed.), where the preface heralds the arrival of matrices
with the following near-apologia:
"A significant change in this edition occurs in the notation used to describe the
operations of multiple regression. Matrix algebra replaces the original summation operators, and a short appendix on matrix algebra is included."


Rao $(1952)$ begins with some thirty
pages of matrices, having written in his preface "The problems of multivariate analysis resolve themselves into an analysis of the dispersion matrix and reduction of determinants."

Also, in $\rm [II],$

For some other purposes, however, it may be important to select a coordinate system so that the variates have desired statistical properties. One might say that they involve characterizations of inherent properties of normal distributions and of samples. These are closely related to the algebraic problems of canonical forms of matrices. An example is finding the  normalized linear combination of variables with maximum or minimum variance (finding principal components); this amounts to finding a rotation of axes that carries the covariance matrix to diagonal form. Another example is characterizing the dependence between two sets of variates (finding  canonical correlations). These problems involve the characteristic roots and vectors of various matrices.

I would urge in general to please have a quick glance of $[\rm I]$ for its brilliant chronological compilation and the history: Searle does a brilliant job there.
I would look into some other materials for relevant quotes and might update it later.

References:
$\rm [I]$ The Infusion of Matrices into Statistics, Shayle R. Searle, Biometrics Unit Technical Reports; Number $\rm BU-1444-M, $ March $1999.$
$\rm [II]$ An Introduction to Multivariate Statistical Analysis, T. W. Anderson, John Wiley & Sons., $2003, $ sec. $1.2, $ p. $3.$
