Which component in my tap water "scale" does not react with acid? Where I live we have pretty hard tap water and so I'm used to scale build-up in kettles and on bathroom tiles, easily removed with acids (it's mostly calcium carbonate I guess). However it seems a part of the "scale" is resistant to acid. For example, I have a glass that I use in my office to drink tap water throughout the day, so the remaining drops of water dry often at the bottom. This results in a circle of "scale", but a significant part of that remains even after treatment with sulfamic acid. Similarly, it seems a part of the residue that forms on my bathroom tiles is not removed by acidic cleaners.

Looking at the water quality charts of my utility, the only part that I think could be responsible just by guessing at the amounts present is $\ce{SiO2}$, at 29 mg/l. Does this form a white, scale-like substance that's hard to remove, at least with acid?
(Tap water analysis here, fold down the Wasserwerk Schulensee part.)

Below an image of my glass after treatment with warm sulfamic acid. The limescale present bubbles off. The remaining white stuff can be scratched off with a knife, so it's actually a residue and not just "etched" glass, but it is pretty tough.
A likely candidate is calcium sulfate, which forms a relatively soft mineral (gypsum), hence removable with a knife, but it has limited solubility in water and does not react with common acids.

See here for more on water hardness.