I’m assuming that by cross-terms you mean off diagonal terms.
First, when talking about off-diagonal terms we clearly mean in a fixed basis. For example, in an eigenbasis of $\rho$ your state has no off-diagonal elements whatsoever.
Second, what you say about measuring clearly holds when performing measurements in that basis ($|0\rangle, |1\rangle$ in your basis). Or more precisely when we measure an observable that has the same eigenbasis.
Coming to the physical meaning of these off-diagonal terms, they are also called ‘coherences’. They are responsible for the interference effects of a quantum particle. In particular those effects that make them sometime look like waves. The prototypical experiment that reveal quantum coherence is the double slit experiment.
The following thought experiment has been put forward by <PERSON> et al. (see [1]) to describe the essence of the quantum coherence and the double slit experiment.
Imagine to prepare the following state
$$ |\Psi\rangle = ( |\phi\rangle_L + e^{i \theta} |\phi\rangle_R)/\sqrt{2} $$
Where $|\phi\rangle_{L,R}$ are Gaussian wavepackets centered around $L, R$ (left, right) with spread much smaller than their separation. This requirement assures that the two states are essentially orthogonal. 976
The left (right) wave packet is prepared with momentum $p$ ($-p$) such that it travels to the right (left). If you measure the position at $t=0$, you will find two Gaussian blobs around $L$ and $R$ and the phase $\theta$ is not observable. From this experiment alone the state cannot be distinguished from a classical state.
Now let evolve the state until the time where the wavepackets collide. At this point it turns out that if you measure position the phase $\theta$ becomes observable! It’s as if the particle interfered with itself, pretty much like a wave.
If you repeat the experiment many times with different phases $\theta$, and every time you measure the position of the particle (for example, the particle hits a screen), you will indeed see a figure of interference forming on the screen.
Added 19/9/22
Mathematical Details
We work in a setting where there is a free particle (not subject to any potential) moving on the real line. Hence the Hamiltonian is $H=a p^2$ ($a=1/(2m)$). If you want to be pedantic the Hilbert space is $\mathcal{H} = L^2(\mathbb{R})$.
First consider as initial state a Gaussian wavepacket centered around $x_0$ moving with momentum $p_0$:
$$ \psi(x)=\frac{e^{-\frac{(x-x_{0})^{2}}{4\sigma^{2}}}}{(2\pi\sigma^{2})^{1/4}}e^{ip_{0}x}. $$
Correctly $|\psi(x)|^2$ is a Gaussian centered around $x_0$ with standard deviation $\sigma$.
The probability distribution of position at time $t$ is
\begin{align} \left|\psi(x,t)\right|^{2} &=\frac{e^{-\frac{(x-x_{t})^{2}}{2\sigma_{t}^{2}}}}{\sqrt{2\pi\sigma_{t}^{2}}} \ x_{t} &=x_{0}+2p_{0} a t \ \sigma_{t} &=\sqrt{\sigma^{2}+\frac{a^{2}t^{2}}{\sigma^{2}}}. \end{align}
This means that if $p_{0}>0$ ($p_{0}<0$) the wavepacket travels to the right (left). Moreover the wavepacket spread a bit in time according to $\sigma_t$.
Now initialize the system in the following superposition. 795