title: Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision
    Transformers?
summary: Kolmogorov-Arnold networks (KANs) are a remarkable innovation consisting of
  learnable activation functions with the potential to capture more complex
  relationships from data. Although KANs are useful in finding symbolic
  representations and continual learning of one-dimensional functions, their
  effectiveness in diverse machine learning (ML) tasks, such as vision, remains
  questionable. Presently, KANs are deployed by replacing multilayer perceptrons
  (MLPs) in deep network architectures, including advanced architectures such as
  vision Transformers (ViTs). In this paper, we are the first to design a general
  learnable Kolmogorov-Arnold Attention (KArAt) for vanilla ViTs that can operate
  on any choice of basis. However, the computing and memory costs of training
  them motivated us to propose a more modular version, and we designed particular
  learnable attention, called Fourier-KArAt. Fourier-KArAt and its variants
  either outperform their ViT counterparts or show comparable performance on
  CIFAR-10, CIFAR-100, and ImageNet-1K datasets. We dissect these architectures'
  performance and generalization capacity by analyzing their loss landscapes,
  weight distributions, optimizer path, attention visualization, and spectral
  behavior, and contrast them with vanilla ViTs. The goal of this paper is not to
  produce parameter- and compute-efficient attention, but to encourage the
  community to explore KANs in conjunction with more advanced architectures that
  require a careful understanding of learnable activations. Our open-source code
  and implementation details are available on: https://subhajitmaity.me/KArAt
paper_authors._id: 67d4f1b1643653fd1cea5b5a
paper_authors.name: Subhajit Maity
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-15T03:19:40.103Z
paper_authors._id: 66d5279130d7ea0b28d6d5d2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66d5279130d7ea0b28d6d5d2/oGFmux--lARBF4PKSkHAu.png
paper_authors.fullname: Subhajit Maity
paper_authors.type: user
paper_authors.user: maitysubhajit
paper_authors._id: 67d4f1b1643653fd1cea5b5b
paper_authors.name: Killian Hitsman
paper_authors._id: 67d4f1b1643653fd1cea5b5c
paper_authors.name: Xin Li
paper_authors._id: 67d4f1b1643653fd1cea5b5d
paper_authors.name: Aritra Dutta
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-15T14:35:55.373Z
paper_authors._id: 67d58d156db0e6f0c33c0f60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67d58d156db0e6f0c33c0f60/9KiFw0iEMZQHcHnm1k0ED.jpeg
paper_authors.fullname: Aritra Dutta
paper_authors.type: user
paper_authors.user: aritradutta
paper_title: Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision
    Transformers?
paper_summary: Kolmogorov-Arnold networks (KANs) are a remarkable innovation consisting of
  learnable activation functions with the potential to capture more complex
  relationships from data. Although KANs are useful in finding symbolic
  representations and continual learning of one-dimensional functions, their
  effectiveness in diverse machine learning (ML) tasks, such as vision, remains
  questionable. Presently, KANs are deployed by replacing multilayer perceptrons
  (MLPs) in deep network architectures, including advanced architectures such as
  vision Transformers (ViTs). In this paper, we are the first to design a general
  learnable Kolmogorov-Arnold Attention (KArAt) for vanilla ViTs that can operate
  on any choice of basis. However, the computing and memory costs of training
  them motivated us to propose a more modular version, and we designed particular
  learnable attention, called Fourier-KArAt. Fourier-KArAt and its variants
  either outperform their ViT counterparts or show comparable performance on
  CIFAR-10, CIFAR-100, and ImageNet-1K datasets. We dissect these architectures'
  performance and generalization capacity by analyzing their loss landscapes,
  weight distributions, optimizer path, attention visualization, and spectral
  behavior, and contrast them with vanilla ViTs. The goal of this paper is not to
  produce parameter- and compute-efficient attention, but to encourage the
  community to explore KANs in conjunction with more advanced architectures that
  require a careful understanding of learnable activations. Our open-source code
  and implementation details are available on: https://subhajitmaity.me/KArAt
paper_ai_keywords: Kolmogorov-Arnold networks (KANs) learnable activation functions multilayer perceptrons (MLPs) vision Transformers (ViTs) Kolmogorov-Arnold Attention (KArAt) Fourier-KArAt loss landscapes weight distributions optimizer path attention visualization spectral behavior