Question: For the 4 jobs that I have worked at, I dressed formal twice, business casual once and more casual once. How important is the dressing code to programming interviews these days? Will the employer or interviewer not take you seriously if you come in casually: plain shirt and khaki pants with casual walking shoes(non-sneakers)?

Answer: 
Some interviewers will not care, so long as your appearance meets the dress standards that they employ on the workplace.

However, many interviewers will care. It is taken as a sign of respect, and of taking a serious interest in the position if you come dressed formally. Conversely, failure to do so may very well be seen as not taking the position seriously, demonstrating a poor understanding of professional expectations, or even disrespectful.

Why take the chance that the interviewer will be picky? Always dress formally, and you can never go wrong.
Unless you show up wearing a white tuxedo, you will almost never have points taken off for being too formal, so why chance it. I would say suit by default, and button up shirt and khakis at an absolute minimum and only if you happen to know that the place is super casual. Always try to out-dress the person interviewing you.