  The increasingly large size of modern pretrained language models not only
makes them inherit more human-like biases from the training corpora, but also
makes it computationally expensive to mitigate such biases. In this paper, we
investigate recent parameter-efficient methods in combination with
counterfactual data augmentation (CDA) for bias mitigation. We conduct
extensive experiments with prefix tuning, prompt tuning, and adapter tuning on
different language models and bias types to evaluate their debiasing
performance and abilities to preserve the internal knowledge of a pre-trained
model. We find that the parameter-efficient methods (i) are effective in
mitigating gender bias, where adapter tuning is consistently the most effective
one and prompt tuning is more suitable for GPT-2 than BERT, (ii) are less
effective when it comes to racial and religious bias, which may be attributed
to the limitations of CDA, and (iii) can perform similarly to or sometimes
better than full fine-tuning with improved time and memory efficiency, as well
as maintain the internal knowledge in BERT and GPT-2, evaluated via fact
retrieval and downstream fine-tuning.

Computation and Language