What is the difference between something being "true" and 'true with probability 1"? In the beginning of chapter 2 of Information Theory, Inference and Algorithms, the author says that he will refrain from being unnecessarily rigorous and provides the example of saying that something is "true" instead of saying more formally that said thing is "true with probability 1".
Is there a difference between something being "true" and 'true with probability 1" ?

If something is true, then it is true with probability of one. Or at least let us assume that without raising further complications. But something with probability of one is not necessarily true. This is the notion of something being almost surely true.
Example
Suppose we sample uniformly on the interval $x \in [0,1] \subset \mathbb{R}$. Since $P(x=\frac{1}{2}) = 0$, we can infer that $P(\lnot [x = \frac{1}{2}]) = 1 - P(x = \frac{1}{2}) = 1$. It is almost surely the case that you will not sample $x=\frac{1}{2}$, but it isn't impossible.
Example: Dart Throwing
This example is from Wikipedia.

Imagine throwing a dart at a unit square (a square with an area of 1) so that the dart always hits an exact point in the square, in such a way that each point in the square is equally likely to be hit. Since the square has area 1, the probability that the dart will hit any particular subregion of the square is equal to the area of that subregion. For example, the probability that the dart will hit the right half of the square is 0.5, since the right half has area 0.5.


Next, consider the event that the dart hits exactly a point in the diagonals of the unit square. Since the area of the diagonals of the square is 0, the probability that the dart will land exactly on a diagonal is 0. That is, the dart will almost never land on a diagonal (equivalently, it will almost surely not land on a diagonal), even though the set of points on the diagonals is not empty, and a point on a diagonal is no less possible than any other point.

