Memory-efficient Energy-adaptive Inference of Pre-Trained Models on Batteryless Embedded Systems
<|reference_start|>Memory-efficient Energy-adaptive Inference of Pre-Trained Models on Batteryless Embedded Systems: Batteryless systems frequently face power failures, requiring extra runtime buffers to maintain inference progress and leaving only a memory space for storing ultra-tiny deep neural networks (DNNs). Besides, making these models responsive to stochastic energy harvesting dynamics during inference requires a balance between inference accuracy, latency, and energy overhead. Recent works on compression mostly focus on time and memory, but often ignore energy dynamics or significantly reduce the accuracy of pre-trained DNNs. Existing energy-adaptive inference works modify the architecture of pre-trained models and have significant memory overhead. Thus, energy-adaptive and accurate inference of pre-trained DNNs on batteryless devices with extreme memory constraints is more challenging than traditional microcontrollers. We combat these issues by proposing FreeML, a framework to optimize pre-trained DNN models for memory-efficient and energy-adaptive inference on batteryless systems. FreeML comprises (1) a novel compression technique to reduce the model footprint and runtime memory requirements simultaneously, making them executable on extremely memory-constrained batteryless platforms; and (2) the first early exit mechanism that uses a single exit branch for all exit points to terminate inference at any time, making models energy-adaptive with minimal memory overhead. Our experiments showed that FreeML reduces the model sizes by up to $95 \times$, supports adaptive inference with a $2.03-19.65 \times$ less memory overhead, and provides significant time and energy benefits with only a negligible accuracy drop compared to the state-of-the-art.<|reference_end|>
@article{farina2024memory-efficient,
    title={Memory-efficient Energy-adaptive Inference of Pre-Trained Models on
  Batteryless Embedded Systems},
    author={Pietro Farina, Subrata Biswas, Eren Y{i}ld{i}z, Khakim Akhunov, Saad
  Ahmed, Bashima Islam, Kas{i}m Sinan Y{i}ld{i}r{i}m},
    journal={arXiv preprint arXiv:2405.10426},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10426},
    primaryClass={cs.LG cs.AI cs.CL}
}