What is the difference between something being "true" and 'true with probability 1"? In the beginning of chapter 2 of Information Theory, Inference and Algorithms, the author says that he will refrain from being unnecessarily rigorous and provides the example of saying that something is "true" instead of saying more formally that said thing is "true with probability 1".
Is there a difference between something being "true" and 'true with probability 1" ?

As Galen indicated, this is the concept of almost surely or almost everywhere.
To provide a more general framework, consider a measure space $(X, \mathfrak A, \mu). $ For a measurable set $A\in \mathfrak A, $ a property $\mathsf Q$ holds almost everywhere on $A$, provided there exists a null set $A_0\subset A$  that is $\mu(A_0) = 0,$ and
$$\mathsf Q(x)~~~ \forall x\in A\setminus A_0.\tag 1$$

Reference:
$[\rm I]$ Real Analysis, H. L. Royden, P. M. Fitzpatrick, Pearson Education, $2010, $ section $2.5, $ p. $45.$
