A More Accurate Approximation of Activation Function with Few Spikes Neurons
<|reference_start|>A More Accurate Approximation of Activation Function with Few Spikes Neurons: Recent deep neural networks (DNNs), such as diffusion models [1], have faced high computational demands. Thus, spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. However, conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions, such as Swish [2]. To approximate activation functions with spiking neurons, few spikes (FS) neurons were proposed [3], but the approximation performance was limited due to the lack of training methods considering the neurons. Thus, we propose tendency-based parameter initialization (TBPI) to enhance the approximation of activation function with FS neurons, exploiting temporal dependencies initializing the training parameters.<|reference_end|>
@article{jeong2024a,
    title={A More Accurate Approximation of Activation Function with Few Spikes
  Neurons},
    author={Dayena Jeong, Jaewoo Park, Jeonghee Jo, Jongkil Park, Jaewook Kim,
  Hyun Jae Jang, Suyoun Lee, Seongsik Park},
    journal={arXiv preprint arXiv:2409.00044},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.00044},
    primaryClass={cs.NE cs.LG}
}