You are missing the use the gradient calculated by the loss (or the next layer). Backpropagation for a Linear Layer 1 by Justin Johnson derives the formula for a simple case and was part of the CS231n lecture.