You have it right, the $V$ function gives you the value of a state, and $Q$ gives you the value of an action in a state (following a given policy $\pi$). I found the clearest explanation of Q-learning and how it works in <PERSON> book "Machine Learning" (1997), ch. 13, which is downloadable. $V$ is defined as the sum of an infinite series but its not important here. What matters is the $Q$ function is defined as
$$ Q(s,a ) = r(s,a ) + \gamma V^{}(\delta(s,a)) $$ where V is the best value of a state if you could follow an optimum policy which you don't know. However it has a nice characterization in terms of $Q$ $$ V^{}(s)= \max_{a'} Q(s,a') $$ Computing $Q$ is done by replacing the $V^$ in the first equation to give $$ Q(s, a) = r(s, a) + \gamma \max_{a'} Q(\delta(s, a), a') $$
This may seem an odd recursion at first because its expressing the Q value of an action in the current state in terms of the best Q value of a successor state, but it makes sense when you look at how the backup process uses it: The exploration process stops when it reaches a goal state and collects the reward, which becomes that final transition's Q value. Now in a subsequent training episode, when the exploration process reaches that predecessor state, the backup process uses the above equality to update the current Q value of the predecessor state. 915
Next time its predecessor is visited that state's Q value gets updated, and so on back down the line (<PERSON>'s book describes a more efficient way of doing this by storing all the computations and replaying them later). Provided every state is visited infinitely often this process eventually computes the optimal Q
Sometimes you will see a learning rate $\alpha$ applied to control how much Q actually gets updated: $$ Q(s, a) = (1-\alpha)Q(s, a) + \alpha(r(s, a) + \gamma \max_{a'} Q(s',a')) $$ $$ = Q(s, a) + \alpha(r(s, a) + \gamma \max_{a'} Q(s',a') - Q(s,a)) $$ Notice now that the update to the Q value does depend on the current Q value. <PERSON>'s book also explains why that is and why you need $\alpha$: its for stochastic MDPs. Without $\alpha$, every time a state,action pair was attempted there would be a different reward so the Q^ function would bounce all over the place and not converge. $\alpha$ is there so that as the new knowledge is only accepted in part. Initially $\alpha$ is set high so that the current (mostly random values) of Q are less influential. $\alpha$ is decreased as training progresses, so that new updates have less and less influence, and now Q learning converges 915