LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
<|reference_start|>LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings: Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors.<|reference_end|>
@article{wang2024llms,
    title={LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with
  LLM Token Embeddings},
    author={Duo Wang and Yuan Zuo and Fengzhi Li and Junjie Wu},
    journal={arXiv preprint arXiv:2408.14512},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.14512},
    primaryClass={cs.LG cs.AI cs.CL}
}