The training loop is a loop over the “training data” that you use to show your neural network what the correspondence between the input and out should be. It is in the context of the whole model, not an individual hidden layer. A real life example of a training loop would be teaching colors to a child – you show a color, and say its name. The child tries to mimick it, and gets it wrong initially. You repeat it again, over and over, until the child starts getting it right. This is what the training loop is.
I am not sure I understand what you mean by that. “All values inside array N go through input x0”? If this is the design you need, you can certainly do it – just write a for-loop, iterating over the array N, and pass its elements into x0 one-by-one. However, from the diagram that I see, x0 is an input for some feature at its appropriate position. You can have any number of elements in that column. Linear algebra will take care of it.
If there is a need to implement the matrix multiplication from scratch, you can just write a loop. If you need an optimized implementation, there is a good explanation in the CLRS book on “Strassen algorithm” that you should read on (Strassen algorithm - Wikipedia). Either way, I don’t really see the point of writing your own matrix multiplication – there are so many libraries that do it very well.
I will be honest here, 2GB memory might not be enough to train a large network. One might argue that if you have a GPU with a lot of memory, you might pull it off, but if it is a 32-bit OS with 2GB RAM, I doubt it has a very powerful GPU. You might be able to train simple networks, but nothing super-fancy. If you need a powerful machine to train a deep network, you can use https://colab.research.google.com. It limits the amount of time you can run the training loop for, but I think it will still give you better results than the machine you currently have.
With that said, although the machine you described might not be good for training, it is perfectly fine for inference. You can download a lot of pretrained models from the PyTorch Hub | PyTorch and use them on that machine.