Yes, you are right that the term $\frac{1}{\pi_\theta(a_t|s_t)}$ itself is favoring the less likely actions. But don't forget how $\nabla \pi_\theta(a_t|s_t)$ is calculated, and you will see that $\nabla \pi_\theta(a_t|s_t)$ is over-emphasizing on the first action and $\frac{1}{\pi_\theta(a_t|s_t)}$ is correcting it.
Recall that the derivative for y =softmax(x) is $\frac{\partial y_i}{\partial x_j} = y_i(\delta_{ij}-y_i)$. 915
Notice that $y_i$ is multiplied in the front, i.e. over-emphasizing the more likely $y_i$. By multiplying $\frac{1}{y_i}$ in the front, the over-emphasizing effect is being corrected.
In your example, when ${\pi_\theta(a_t|s_t)}=[0.5, 0.1, 0.1, 0.1, 0.1, 0.1]$, we can write down that $$\nabla \pi_\theta(a_t|s_t)=\left(\begin{array}{cccccc} 0.5\times0.5 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1\ -0.5\times0.1 & 0.1\times0.9 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 \ -0.5\times0.1 & -0.1\times0.1 & 0.1\times0.9 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 \ -0.5\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9 &-0.1\times0.1 & -0.1\times0.1 \ -0.5\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9 & -0.1\times0.1 \ -0.5\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9\ \end{array}\right)$$ And by multiplying it with $\frac{1}{\pi_\theta(a_t|s_t)}=[\frac{1}{0.5},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1}]$, you effectively get $$\nabla \log\pi_\theta(a_t|s_t)=\frac{1}{\pi_\theta(a_t|s_t)}\nabla \pi_\theta(a_t|s_t)= \left(\begin{array}{cccccc} 0.5 & -0.5, &-0.5, &-0.5, &-0.5, &-0.5, \ -0.1 & 0.9 & -0.1 & -0.1 & -0.1 & -0.1 & \ -0.1 & -0.1 & 0.9 & -0.1 & -0.1 & -0.1 & \ -0.1 & -0.1 & -0.1 & 0.9 & -0.1 & -0.1 & \ -0.1 & -0.1 & -0.1 & -0.1 & 0.9 & -0.1 & \ -0.1 & -0.1 & -0.1 & -0.1 & -0.1 & 0. 374