bird-of-paradise

/

deepseek-mla

like 7

Hi Hudson! Great question about the matrix absorption optimization.

You're absolutely right that the paper mentions these matrices can be absorbed during inference for efficiency. In this implementation, I kept them separate for a few reasons:

1. Clarity and Understanding: This implementation prioritizes making the architecture clear and understandable, showing each component explicitly as described in the paper's equations.

2. Training vs Inference: The absorption optimization is specifically for inference. During training, you'd still need the separate matrices.

3. Flexibility: Keeping them separate allows for easier experimentation and modification of individual components.