Can reinforcement learning directly work with advantage function? When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only need the advantage function A(s, a) that describes the relative future reward for these actions, instead of the Q-function Q(s, a). Then is it possible to write Q-learning in such a way that it only involves the advantage function? By reading the paper on double DQN my impression is we need to compute both advantage function A(s, a) and value function V(s) explicitly instead of just the advantage function, but I don't see why that is the case.


When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only need the advantage function A(s, a) that describes the relative future reward for these actions, instead of the Q-function Q(s, a).

This is true for determining a current policy. However, this doesn't cover estimating the value function that you want to use from experience.

Then is it possible to write Q-learning in such a way that it only involves the advantage function?

You need an update mechanism for the value functions that can process experience and update the estimates for the value functions. Q learning is based on TD learning, which is derived from the Bellman equations for the value functions.
There is no simple Bellman equation for the advantage function $A(s,a)$ which is separate from $V(s)$ and $Q(s,a)$. You could write out a derivation for $A(s,a)$ that doesn't mention $V(s)$ or $Q(s,a)$ by name, but all the calculations necessary to estimate one or other or both will be inside it.
You may be able to design a solution that avoids explicitly calculating and maintaining $Q(s,a)$, but in that case you will not be able to avoid calculating $V(s)$.
