Llama from scratch, or how to implement a paper without crying
The TL;DR pointers are really great, and the note about asserting the shape of tensors applies any common linear algebra library out there, as far as I know. When working on complex LA code, it's <i>extremely</i> important to take small steps and code defensively. In my opinion programming linear algebra with any mainstream language is absolutely <i>atrocious</i> due to lack of compile-time checking of tensor shapes, which should properly be part of a tensor's type and would make it impossible to compile if you're trying to multiply a 3x4 by a 3x4 without transposing first. It really, really sucks to run a long calculation only to fail on an operation due to mismatched dimensions.<p>IMO PyTorch tensors should also have their device statically typed; right now you get a run-time error if you try to multiply a tensor in CPU memory by one in GPU memory.