What is the difference between something being "true" and 'true with probability 1"? In the beginning of chapter 2 of Information Theory, Inference and Algorithms, the author says that he will refrain from being unnecessarily rigorous and provides the example of saying that something is "true" instead of saying more formally that said thing is "true with probability 1".
Is there a difference between something being "true" and 'true with probability 1" ?

This difference occurs because of the difference between probability and possibility
This answer is based on a longer paper O'Neill (2014) that looks at the interaction of a binary possibility operator and a probability measure.  Have a look at this paper if you'd like a more detailed exposition of the subject covered in this answer.  You can also find a related answer here.
The basic idea here is that we can have a space of outcomes $\Omega_*$ that are all possible, but some outcomes in the space may be so improbable that we can remove them from consideration while still having probability one for the set of remaining outcomes.  For example, if we have a continuous random variable $X$ with support $\mathscr{X}$ then every value in the support is possible, but each specific value also has zero probability of occurring.  Thus, if we take some value $x \in \mathscr{X}$ then we know that $x$ is a possible value, but we also have $\mathbb{P}(X \neq x) = 1$.  In this case, it is not necessarily true that $X \neq x$ (since $x$ is a possible value) but it is true with probability one (we say that this occurs "almost surely").  You can easily construct examples of the divergence between possibility and probability by using zero-probability events for continuous random variables.  The dart-throwing example cited in the other answer here is one instance of this.
This split occurs because of the difference between probability and possibility, and it is something you need to keep in mind when dealing with continuous random variables or in other similar cases.  Below I will give you a basic outline of how possibility is represented in the analysis of events, and how it diverges from probability.  There is a sizable literature analysing mathematical representations of possibility (this field is known as possibility theory), which is closely related to modal logic and fuzzy set theory.  Overviews of this field can be found in Yager (ed) (1982), Kacprzyk and Orlovski (eds) (1987), Dubois and Prade (1993), Terano, Asai and Sugeno (eds) (1992), Zadeh and Kacprzyk (1992) and Dubois (2006).

A Brief Description of Possibility Theory
The field of possibility theory is formalised in a similar manner to the field of probability theory.  In both cases we have an overall space of outcomes $\Omega$ and a class of events $\mathscr{E} \subseteq \Omega$ on that space.  Possibilty theory works with a set operator that measures events on a scale from zero to one, just as in probability theory.  The theory is characterised by a possibility function $\nabla$ that obeys a set of axioms that are close to those of probability theory, except that they display different behaviour when applied to unions of disjoint events.  The general theory allows for a possibility measure that takes on values between zero and one, but here I will use an all-or-nothing possibility measure taking on values zero or one.  (This is partly for illustrative purposes and partly because this special case conforms best to the intuitive idea of possibility.)  The properties characterising a possibility measure $\nabla$ and a probability measure $\mathbb{P}$ are as follows (in the last line we refer to a countable set $\mathscr{E}_1,\mathscr{E}_2,\mathscr{E}_3,...$ of disjoint events):
$$\begin{matrix}
\text{Possibility measure} \quad \quad & & & & & \text{Probability measure} \quad \\[6pt]
\nabla(\varnothing) = 0 \quad \quad \quad & & & & & \mathbb{P}(\varnothing) = 0 \quad \quad \\[6pt]
\nabla(\Omega) = 1 \quad \quad \quad & & & & & \mathbb{P}(\Omega) = 1 \quad \quad  \\[6pt]
\nabla ( \bigcup_i \mathscr{E}_i ) = \max_i \nabla(\mathscr{E}_i) & & & & & \mathbb{P} ( \bigcup_i \mathscr{E}_i ) = \sum_i \mathbb{P}(\mathscr{E}_i) \\[6pt]
\end{matrix}$$
As you can see, both functions give zero measure for the empty event $\varnothing$ and unit measure for the full space $\Omega$.  However, when it comes to dealing with countable unions of disjoint sets, the two measures differ --- the probability of a countable union of disjoint events is equal to the sum of the probabilities of those events, whereas the possibility measure of a countable union of disjoint events is equal to the maximum of the possibility measures of those events.
The above measure for possibility might look a bit strange, since we are used to thinking of possibility as a binary.  The possibility measure is formulated in a more generalised way (allowing any value between zero and one) in order to accommodate fields like fuzzy logic and modal logic.  Nevertheless, for present purposes, we can simplify possibility theory to deal only with events classified using the binary categorisation (zero or one) with $\nabla(\mathscr{E}) = 0$ meaning that $\mathscr{E}$ is impossible and $\nabla(\mathscr{E}) = 1$ meaning that $\mathscr{E}$ is possible.  (Similarly, we say that an event is certain if its negation is impossible.)  The third property above then reduces to saying that the union of a countable set of disjoint events is possible if and only if at least one of the events is possible, which corresponds to our intuition.
(In fact, if all events have possibility measure zero or one then it is possible to extend the properties for the possibility measure to allow the above maximisation property over arbitrary unions, not just countable unions.  This extension is not generally possible for the probability measure, as is well known.)

Relating Probability and Possibility
In order to relate probability and possibility measures on the same space (and thereby see how they interact) we can begin by taking some overarching space $\Omega$ and considering the set of all possible outcomes defined by   $\Omega_* \equiv \{ \omega \in \Omega | \nabla(\{\omega\}) = 1 \}$.  We would then usually begin our analysis with the axiom that the set of all possible outcomes has probability one (i.e., that $\mathbb{P}(\Omega_*)=1$).  Starting with this axiom we can then establish that all certain events are almost sure and all events with positive probability are possible.  However, the converses are not necessarily true --- an almost sure event is not necessarily certain and a possible event does not necessarily have positive probability.
I will leave it as an exercise to you to see if you can derive the above rules from the axiom above.  It is relatively simple to formulate an example using a continuous random variable where you have events that are uncertain but have probability one.
