Not quite---the rule says (according to the BBC article on this) that the four years starts from <i>completion</i> of the building, which would prevent your scenario.  The current judge appears to be arguing that the bales of hay, which were not structural and to my understanding didn't even touch the house, were integral and that the house was not 'complete' until they were removed.  It's a bit of a stretch, and it sure sounds like the judge is just annoyed that this guy found a loophole.