Question: For the 4 jobs that I have worked at, I dressed formal twice, business casual once and more casual once. How important is the dressing code to programming interviews these days? Will the employer or interviewer not take you seriously if you come in casually: plain shirt and khaki pants with casual walking shoes(non-sneakers)?

Answer: 
Some interviewers will not care, so long as your appearance meets the dress standards that they employ on the workplace.

However, many interviewers will care. It is taken as a sign of respect, and of taking a serious interest in the position if you come dressed formally. Conversely, failure to do so may very well be seen as not taking the position seriously, demonstrating a poor understanding of professional expectations, or even disrespectful.

Why take the chance that the interviewer will be picky? Always dress formally, and you can never go wrong.
My general rule of thumb is dress as though you were meeting a client....b/c you are. Your product is yourself and your skill set.

Unless they tell you that the interview will be 'causal' go full suit and make sure you look neat and professional. If they do say causal I would even suggest going business causal to the interview.