txt is used as part of the Robots Exclusion Standard, a voluntary protocol the Internet Archive respects that disallows bots from indexing certain pages delineated by its creator as off-limits.
What kind of protocol is the Robots Exclusion Standard?
voluntary