Machine Learning, Software Development, Artificial Intelligence, Technology, Data Science.
that your weights move only within a tiny subspace. But you can avoid this by applying LoRA iteratively; in each iteration you a) fuse the low-rank matrices into the frozen weights and b) create new low-rank matrices. With rank R subspaces and N LoRA restarts, you could in principle move in a