  Recent years have witnessed the great success of Graph Neural Networks (GNNs)
in handling graph-related tasks. However, MLPs remain the primary workhorse for
practical industrial applications due to their desirable inference efficiency
and scalability. To reduce their gaps, one can directly distill knowledge from
a well-designed teacher GNN to a student MLP, which is termed as GNN-to-MLP
distillation. However, the process of distillation usually entails a loss of
information, and ``which knowledge patterns of GNNs are more likely to be left
and distilled into MLPs?" becomes an important question. In this paper, we
first factorize the knowledge learned by GNNs into low- and high-frequency
components in the spectral domain and then derive their correspondence in the
spatial domain. Furthermore, we identified a potential information drowning
problem for existing GNN-to-MLP distillation, i.e., the high-frequency
knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency
knowledge during distillation; we have described in detail what it represents,
how it arises, what impact it has, and how to deal with it. In this paper, we
propose an efficient Full-Frequency GNN-to-MLP (FF-G2M) distillation framework,
which extracts both low-frequency and high-frequency knowledge from GNNs and
injects it into MLPs. Extensive experiments show that FF-G2M improves over the
vanilla MLPs by 12.6% and outperforms its corresponding teacher GNNs by 2.6%
averaged over six graph datasets and three common GNN architectures.

Machine Learning