Is sampling with replacement better than sampling without replacement? This question is more specific to machine learning. Is sampling with replacement good for random forests because it leaves some out of bag samples for testing or is it because it creates datasets/decision trees that have more variation? Or is it a mixture of both the facts.

We use sampling with replacement because we use bootstrap. Bootstrap imitates how we sampled the data from the population. When sampling with replacement, we end up with a sample of the same size as your original data. What bootstrap does by this, is it lets you imitate the data generating process, the underlying distribution of the data, and the variability. When sampling without replacement you would either use smaller samples or just permute the data, assuming that it is independent and identically distributed, this would lead to the same sample.
