robots.txt is a like a gentlemen's agreement and bad actors don't care about respecting it. But maybe a regulation that consider it (or a similar technical solution) as a binding agreement is something that is missing for society. (as well as for browser and OS settings)