When a piston cylinder system that evolves isothermally and quasistatically is irreversible? Suppose we have two states: $\ce{A}$ and $\ce{B}$.
Considering a quasi-static isothermal process of an ideal gas, (and another condition, that is $T = T_\mathrm{surr}$) that join those states.

Then the Poisson's relation $P_\ce{A}V_\ce{A} = P_\ce{B}V_\ce{B}$ is fulfilled.
that means after integrating

$$W = nRT\ln\left(\frac{V_\ce{B}}{V_\ce{A}}\right)$$

$$\mathrm{d}S_\mathrm{s} = -\mathrm{d}S_\mathrm{surr}$$

because $T = T_\mathrm{surr}$. Then

$$\mathrm{d}S_\mathrm{u} = 0.$$

So my guess is that no, that irreversibilities on those conditions come only on finite temperature difference with surroundings.

But then I remember friction. If piston with friction has $x$ joules lost by friction, how does that change everything?

I need someone to "shed light" on this issue. There is always a lot of confusion when we consider a quasi-static but irreversible process.
If it is done quasi statically with friction involved and, if the system is in contact with a constant temperature bath at the initial system temperature T, then, for the same volume change, the change in entropy of the system will be the same.  That means that all the entropy generated as a result of frictional irreversibility will be transferred to the surroundings (the constant temperature bath).  This will be equal to the frictional work divided by the temperature T.  So the entropy of the universe will increase by this amount.