The folks we elect routinely refuse to tie specific revenues collect to specific outlays, like the affordable housing you are looking for.  Any money they collect goes into general revenues to spend as they see fit.

If revenues and outlays were tied more closely, as you suggest, it would likely have the effect of reducing tax evasion, because many of the folks evading or avoiding tax are doing so because they don't trust what government does with their money.