a difference in the online inference setting where the batch size is typically as small as one. In a
generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al., b)
medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a
very small bottleneck dimension (Table 1).
This problem gets worse when we need to shard the model as done in Shoeybi et al. (2020); Lep-
ikhin et al. (2020), because the additional depth requires more synchronous GPU operations such as
AllReduce and Broadcast, unless we store the adapter parameters redundantly many times.
LORA