The Planning Inspectorate has backed the council's decision to demand the property near Crymych is demolished.
Owner's Charlie Hague and Megan Williams argued that the straw-walled house had low impact on the environment.
But inspectors said that did not change rules on building in rural areas.
"There is a lack of proper justification for the benefits of the low-impact development in this case for this matter to be given sufficient weight and to outweigh the policies which seek to control development in the countryside," ruled the planning inspector, Iwan Lloyd.
The owners had been living in a caravan for four years, before moving into the new home just before their son Eli was born.
Sculptor Mr Hague spent more than a year creating the single-storey dwelling from local materials.
The couple were helped by friends and relatives at the neighbouring Lammas eco village.
But Pembrokeshire council began enforcement proceedings earlier this year, after ruling it was an unauthorised dwelling built in open countryside.
Deputy council leader Rob Lewis said he welcomed the planning inspector's decision.
"Planning is a rigorous process with clear guidelines and if it is to be enforced fairly must also be upheld by everyone," he said.
The enforcement notice gives the couple two months to demolish the property.
However, the council will now consider a retrospective planning application for the house.
"It will be considered on its merits," said Mr Lewis.

Aarolina Rutkowska, 35 and of Cross Farm Road in the Peasley Cross area of St Helens, was found not guilty of murder after his trial at Liverpool Crown Court.
Mireslaw Kowalczuk, 29, was fatally stabbed in the chest at a house on Cross Farm Road on 27 December.
He died in the ambulance on his way to hospital.

The High Commissioner of India laid a wreath of marigolds at the Arch of Remembrance in Leicester.
Meanwhile, the British High Commissioner did the same at a ceremony at the India Gate in New Delhi.
Hundreds of people attended the events in memory of the 1,174 Indian Labour Corps men who supplied the army with weapons, food and built barracks.
Updates on this story and more from the East Midlands
The commemorative events were part of The Unremembered project to honour the courage and sacrifice of labourers in World War One.
In Leicester, about 300 school children and dignitaries including the city and Lord mayors held up the names of the men and tied messages to the fallen during the ceremony.
Teacher Mrs Hobson, of Moat Community College, said: "This event is important because it's about our shared heritage... and it allows us to link our students to their Indian heritage."
Squadron Leader Rana Chhina said Indians in Leicester "should be proud" of the connection because it "helps strengthen the bond" between the UK and India.
British architect Sir Edwin Lutyens designed both war memorials in the 1920s.
The workers' names are engraved on the India Gate, which was built in 1931 and took about 10 years to complete.
Work began on the Arch of Remembrance, in Victoria Park, which is half the size of the gate, in 1923 and was finished in 1925.
In 2015, all 44 of Sir Edwin's World War One memorials in England became listed by Historic England.
A Pembrokeshire couple who built a 'hobbit-style' eco home without planning consent have lost a demolition appeal.
[NEXT_CONCEPT]
A woman has been cleared of murdering her partner who was found with stab injuries.
[NEXT_CONCEPT]
War memorials in the UK and India have been twinned to remember Indian workers who died in World War One.