  We introduce a class of permutation centralizer algebras which underly the
combinatorics of multi-matrix gauge invariant observables. One family of such
non-commutative algebras is parametrised by two integers. Its Wedderburn-Artin
decomposition explains the counting of restricted Schur operators, which were
introduced in the physics literature to describe open strings attached to giant
gravitons and were subsequently used to diagonalize the Gaussian inner product
for gauge invariants of 2-matrix models. The structure of the algebra, notably
its dimension, its centre and its maximally commuting sub-algebra, is related
to Littlewood-Richardson numbers for composing Young diagrams. It gives a
precise characterization of the minimal set of charges needed to distinguish
arbitrary matrix gauge invariants, which are related to enhanced symmetries in
gauge theory. The algebra also gives a star product for matrix invariants. The
centre of the algebra allows efficient computation of a sector of multi-matrix
correlators. These generate the counting of a certain class of bi-coloured
ribbon graphs with arbitrary genus.

High Energy Physics - Theory;Group Theory;Representation Theory