Nested loops are expected to be slow.
You could use an im2col approach and matrix multiplications for the manual convolution approach.
What id your use case for this code?
If it’s a demo code to show the underlying operations of conv layers (to students) I wouldn’t care too much about the performance.