Wokingham Borough Council served a High Court injunction on travellers for setting up "unauthorised" pitches and an access road on the private land.
The Gypsy Council said the authority had been quick to get an injunction but refused to provide enough gypsy sites.
The council said it "utterly refutes" the accusation of racism.
Housing applications in the area had been judged acceptable against the borough's planning policies, drawn up following a statutory process including consultation, it said.
Caravans arrived on the Berkshire site on 24 April with the permission of the landowner, who is part of the gypsy community.
But the council has taken legal action because developments that have since taken place do not have planning consent.
It obtained the injunction to prevent further caravans and development on the Berkshire site, after its deadline to exit the site by Monday 27 April was ignored.
"This is not a site allocated for development in the council's local plan," said director of environment, Helen Thwaites.
"It is located within the countryside where the council's policies seek to protect the rural character."
Residents near the site, in Blagrove Lane, have also expressed concern.
Ian Andrews, of Evendons Area Residents' Association, said there was unhappiness at "a total disregard for the planning process in a quiet rural location".
"Hedgerows have been ripped out during nesting season and more than 50 giant lorry loads of rubble have been dumped into what was the flood plain for several hundred homes," he said.
Residents had also expressed fear of a rise in crime, which Thames Valley Police said was "unfounded".
Joseph Jones, of the Gypsy Council, said: "Councils need to fulfil their target for gypsy community site provision.
"Local authorities would much rather refuse planning permission and take legal action than tackle what they perceive to be the thorny issue of gypsy site provision.
"The council can merrily grant planning permission for houses in the next field, but when a gypsy site application comes in from a gypsy family for a small number of pitches then there's hell to pay.
"That's racism."

Vancouver Aquarium said it had been contacted by the girl's family after staff had raised concerns about "seal finger", which can lead to amputations.
The girl suffered a 10cm (4in) wound when the animal grabbed her with its jaws, her father told CBC News.
He rejected claims that she had been trying to feed the marine mammal.
The video of her being pulled by the dress from the dockside in Richmond, British Columbia, and then rescued by her grandfather, has been widely shared on social media, prompting some users to suggest the family had  been trying to feed sea lions.
"There was somebody beside them that was trying to feed them," said the father, who gave only his surname, Lau, in order to prevent the identification of the girl.
But he admitted his daughter had got too near when she moved forwards to have a closer look.
"That's a lesson she took and she has taken that lesson in a hard way," he said.
Aquarium spokesman Deana Lancaster told ABC news the family had got in touch after staff gave interviews about the risks of "seal finger" infection.
Bacteria living in the mouths of seals and sea lions can cause the painful and potentially serious condition, with symptoms including cellulitis and debilitating joint inflammation.
It is mainly known historically for affecting sealers' hands, although it can travel to any part of the body.
Ms Lancaster said the girl was now going to get the right treatment.
Mr Lau confirmed the family had been in touch with the aquarium and praised the girl's grandfather's quick response.
"If he had a one- or two-second doubt about that, my girl could have been gone by then. That reaction makes him a hero," he said.
He said the family had been "disturbed" by the video going viral, but were mainly just relieved the girl was safe.
"Only thing I care is: God, she is safe," said Mr Lau.
"I could have gone organising a funeral by now rather than doing an interview."
A council has been accused of racism for refusing planning permission for a gypsy camp after approving a housing estate in a field next door.
[NEXT_CONCEPT]
A girl dragged off a dock in Canada by a sea lion is being treated with antibiotics because of the risk of a dangerous infection.