So I think there are two questions here: 1) What are the dominant contributions to the observed CMB Dipole and 2) What can these measurements tell us about the universe.
For (1), as you stated, the dominant contribution is the doppler shift relative to the CMB rest frame. Our movement results in a hot spot in the direction we are moving towards, and a cold spot in the opposite direction. There is also a contribution from the primordial fluctuations, as these fluctuations occur across all scales (with the sourcing amplitude determined by the $A_s$ and power-law slope by $n_s$).
For (2), the meaning of anisotropies is tied up in their relative error. At low $\ell$ there is significant error due to galactic dust as well as "cosmic variance." Cosmic variance is a fundamental limit in our measurements of these very large scale fluctuations since we only have one universe to study; if we only can ask one person in a country we would have a very hard time figuring out the average age of people, but if we can ask 10,000 we could have a very good estimate of the average and whole distribution of possible ages in that country. In the case of the primordial CMB dipole we are only measuring one single mode in one universe, so it is susceptible to huge error and so it isn't particularly "relevant" for CMB analysis.
The plot you show is very old, where the dominant source of error comes from noise in the map itself; measuring very cold things (i.e. 562
2.72 K) is very hard when your detector generates heat and warms up what you are trying to observe! More modern measurements are dominated by noise from foregrounds (i.e. dust and galaxies), as well as cosmic variance over most of the range of interest. Here is a more modern version from the Planck Sattilite:
The green shaded region represents the cosmic variance region; i.e. for the same cosmological parameters we would expect the CMB anisotropies to vary within this region. While not shown explicitly, you can extrapolate to see that $\ell = 1$ has huge error purely from this variation. 675