About half a mile from the palace in which they lived there stood a castle , which was uninhabited and almost a ruin , but the garden which surrounded it was a mass of blooming flowers , and in this garden the youngest Princess used often to walk .