Can gauge fields be avoided by not specifying a coordinate system?
I've been studying quantum field theory, and recently my focus has been on the complex scalar field $\phi(x_\mu)\in\mathbb{C}$ because it seems to be the simplest field that can couple to an interaction like electromagnetism. In particular, I'm interested in seeing how the 4-potential manifests as the gauge field necessary for keeping the Lagrangian invariant under a local phase shift $\phi\to e^{-i\alpha(x)}\phi$.
However, I am having difficulty understanding why we want this local gauge invariance in the first place. A global symmetry makes sense to me because a uniform rotation of every vector has no physical significance. However, rotating the field by a different amount at each point through a local transformation seems like it would produce a completely different field. I am having trouble seeing how this transformation does not change all of the physics. I have been reading through textbooks like <PERSON>, Henneaux & Teitelboim, and <PERSON>, so things are starting to take shape, but I wanted to ask to see whether I was on the right track.
I really like <PERSON> perspective of gauge transformations as changes of coordinates. He talks about how "coordinate systems identify geometric or combinatorial objects with numerical (or standard) ones". 298
Connecting to my focus on the complex scalar field, he writes:
to convert a position on a circle to a number (modulo multiples of 2\pi), or vice versa, one needs to pick an orientation on that circle, together with an origin on that circle. Such a coordinate system then equates the original circle to the standard unit circle
To me, this means that in order to assign a complex number to each point we must choose a reference direction within the plane to call "1"/the positive real line. This is an arbitrary (smooth) assignment at each point in spacetime that should have no physical meaning, so the the physics, i.e. the Lagrangian, should be invariant under an alternative choice of direction. However, all choices of directions for the positive real line are related to each other by a rotation in the complex plane, so the invariance required is really that of a local phase shift. This would make sense to me. Just thinking about putting a complex number everywhere in space seemed simple, but I had not realised the nuances of how that is not possible without implicit choices being made that bring in a the need for a gauge to compensate.
That being said, could this be avoided by not specifying a coordinate system at all? For example, rather than have $\phi\in\mathbb{C}$ could we have $\phi\in V$ for some 2-dimensional vector space $V$? Or perhaps there is another way to have coordinate invariance? The first approach seems problematic, as we'll want to do calculus with the field, so it will need to be described with some smooth manifold. However, differential geometry is meant to be coordinate free and does not introduce gauge fields in order to do calculus. 298