Can a neural network with only $1$ hidden layer solve any problem? Can a network with a single hidden layer of $N$ neurons (where $N\le\infty$) approximate any arbitrary function so that this network’s error approaches $0$ as $N$ approaches $\infty$?

The other answers have already mentioned the Universal approximation theorem and that, in strictly mathematical sense, it does not apply to all the functions and it approximates only to the predefined level of error.
However, there is a different way to interpret the question: if a single layer network can approximate any function (satisfying criteria of the UAT) to any required degree of precision, why do we use multilayer networks? The answer is that, given a specified precision, the single layer network may have to be very big and/or may take a very long time to train. Multilayer networks can be smaller and can be trained faster (provided that we use appropriate techniques, such as the backpropagation algorithm).
