That is pretty much it! Although the one-hot-encoding is more likely to be “word-level” than “character-level”, but I might be wrong.
Anyway, in the case of stock prices, you indeed have a sequence of dense vectors already (I guess). You cannot represent each vector by an index and, therefore, you need a y = Ax transformation, as you said.
But nn.Embedding doesn’t do that; nn.Linear does it! So just go ahead and use that