Question: For most ML models we say they suffer from high bias or high variance, then we correct for it. However, in DL do neural networks suffer from the same concept in the sense that they initially have high bias or high variance and then you correct through regularization and/or dropout? I would argue they initially suffer from high variance and they overfit the data. Then you correct through regularization, add dropout, image pre-processing in the case of CNNs, etc. Is this train of thought correct?

Answer: 
In general NNs are prone to overfitting the training set, which is case of a [high variance](https://datascience.stackexchange.com/questions/45578/why-underfitting-is-called-high-bias-and-overfitting-is-called-high-variance). Your train of thought is generally correct in the sense that the proposed solutions (regularization, dropout layers, etc.) are tools that control the bias-variance trade-off.
Neural networks, including DNNs, don't *by themselves* suffer from high variance any more than other machine learning algorithms. It is just that we find it easier to start the training with more complex networks and control for variance by the techniques you mentioned, than to start with simpler (less expressive) networks and gradually increase their complexity. By observing the network behaviour during training you can already get hints regarding simplification. If you'd start with a simpler network, you'd be tapping in the dark how to augment it.