Unseen Servants can't pick anything up, it can only carry things that you hand to it.  So... if it wasn't really big debris, it could carry it, but you'd have to pick it up and hand it over.  Basically, the only force it exerts is holding things and following you around.