Question: For the 4 jobs that I have worked at, I dressed formal twice, business casual once and more casual once. How important is the dressing code to programming interviews these days? Will the employer or interviewer not take you seriously if you come in casually: plain shirt and khaki pants with casual walking shoes(non-sneakers)?

Answer: 
**Short answer: Unless stated explicitly otherwise, always formal.** 

It is not about being taken seriously, it is about *interviewee* showing respect to the *interviewers* and taking the job application process seriously.
Unless you show up wearing a white tuxedo, you will almost never have points taken off for being too formal, so why chance it. I would say suit by default, and button up shirt and khakis at an absolute minimum and only if you happen to know that the place is super casual. Always try to out-dress the person interviewing you.