Can Mean Square Error cause underfitting? Mean Square Error (MSE) is used in Regression problems to compute the error in prediction. Large errors have a large influence on the MSE, and small errors have almost negligible influence on the error. So if an observation is giving a large error, there's a possibility that the observation is an outlier. And MSE will work really hard to minimize that error. So, does minimize MSE cause Underfitting?

By itself, no. The choice of loss function depends on your data and the nature of the problem you are trying to solve. As you noticed, mean square error is sensitive to large errors, while something like mean absolute error is less sensitive. Such sensitiveness is sometimes a desirable property, while in other cases you need a robust loss function that is insensitive. You use squared error when you need it to be sensitive.
No matter what loss you choose, overfitting is about the whole model, not only the loss. For example, if you use a simple regression model that minimizes the squared error, it has no chance to overfit, because it is not expressive enough. On another hand, if you use instead something like $k$NN no matter of the loss, it could overfit. Basically, if the model can drag the training error to zero (e.g. model with enough parameters to memoize the data), it eventually will, no matter what the loss is.
