Machine Learning, Software Development, Artificial Intelligence, Technology, Data Science.
subspace of rank R*N. To get this to work, you have to re-warmup the learning rate and reset the optimizer state at the start of each iteration. Otherwise you diverge. In practice, this method moves through a higher-volume subspace than just doing LoRA once, but not as high-volume as full