It strikes me that not using a "filter" like the party system would mean our politicians would be spending so much time responding to and trying to win favour with voters that they would not actually have time to think about the sorts of things we need to them to do to provide effective leadership and/or realistic alternative options. What you say may sound good but it makes no sense in the real world.