Patients miss about 10m GP appointments and 5m practice nurse appointments each year, a survey suggests.
Not all patients turn up for their appointment
The Developing Patient Partnerships survey suggests the missed GP appointments alone cost the NHS £180m.
The GPs questioned said forgetfulness was the main cause and 70% said they would consider removing persistent offenders from their lists.
But an organisation which insures doctors warned GPs this should be the last resort.
Two-thirds of the 683 surgeries surveyed said patients should be charged for missing an appointment.
Most doctors who responded to the survey said missed appointments increased waiting times, wasted NHS money and resources and harmed GPs' ability to meet primary care targets.
Doctors believe simple forgetfulness is the main reason why people fail to turn up for their appointment.
However, some blamed patients feeling better, and deciding not to attend.
Dr Terry John, a spokesman for the DPP, said: "These results are disappointing especially if we consider that each GP appointment costs £18.
"On the bright side, most practices believe that this figure can be reduced through effective communication about the impact of missed appointments."
Jayne Tabor, of the Institute of Healthcare Management, said: "Missed appointments impact on all of us.
"Almost everyone is likely to have difficulties booking an appointment to see their GP at some point.
"These difficulties are exacerbated by missed appointments. Cancelling unwanted appointments frees them up for other patients who need them."
A Department of Health spokeswoman said: "Missed appointments matter. They waste GPs' time and NHS' - and so taxpayers' - resources.
"Patients are right to expect a modernised responsive and accessible service.
"But for their part, patients have a responsibility to keep or cancel appointments."
Liberal Democrat health spokesman Steve Webb said: "Serial offenders who persistently fail to turn up or cancel their appointment need to realise the impact their actions are having on other people. However, the patients are not always to blame.
"Inflexible government targets have led to inconvenient appointment systems and patients being unable to get through to the surgery to cancel an appointment."
But Dr Hugh Stewart, a medico-legal adviser for the Medical Defence Union, warned: "While we understand the considerable disruption that missed appointments can cause, we advise GPs to think very carefully before removing patients from practice lists solely because they have missed appointments.
"In most circumstances, removing a patient from a list should be a last resort, to be used when all other options have been exhausted.
"In our experience, the act of removal itself can often lead to a complaint."