Large Language Models, Llm, AI, Deep Learning, Machine Learning.
Problem: Though random permutation is common to use this method, it is not a suitable choice for llms. Gradient vectors have extremely large dimensionality (for e.g. the length of 7B for lamma-2 7b), it results in hectic as it scales. solution: Random Shuffling. It’s suitable for large vectors.