DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models
<|reference_start|>DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models: Large language models have repeatedly shown outstanding performance across diverse applications. However, deploying these models can inadvertently risk user privacy. The significant memory demands during training pose a major challenge in terms of resource consumption. This substantial size places a heavy load on memory resources, raising considerable practical concerns. In this paper, we introduce DP-MemArc, a novel training framework aimed at reducing the memory costs of large language models while emphasizing the protection of user data privacy. DP-MemArc incorporates side network or reversible network designs to support a variety of differential privacy memory-efficient fine-tuning schemes. Our approach not only achieves in memory optimization but also ensures robust privacy protection, keeping user data secure and confidential. Extensive experiments have demonstrated that DP-MemArc effectively provides differential privacy-efficient fine-tuning across different task scenarios.<|reference_end|>
@article{liu2024dp-memarc:,
    title={DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient
  Language Models},
    author={Yanming Liu, Xinyue Peng, Yuwei Zhang, Xiaolan Ke, Songhang Deng,
  Jiannan Cao, Chen Ma, Mengchen Fu, Xuhong Zhang, Sheng Cheng, Xun Wang,
  Jianwei Yin, Tianyu Du},
    journal={arXiv preprint arXiv:2406.11087},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11087},
    primaryClass={cs.CR cs.AI cs.CL cs.LG}
}