How does Mac OS keep track of moved files? An application I was using had a .dockset file stored in my home directory. I moved this file, and the application automatically updated the location of the file. I notice a similar effect when moving files that I'm currently editing in apps like Xcode, TextEdit, BBEdit, Preview, iWork, etc.

How does Mac OS keep track of these file movements?
Most Apps don't use the Bookmarks (mentioned by Matthieu) for this purpose, because Bookmarks are meant to locate a file after the app has been quit in between), but rather either keep the file open while the app runs, or reference the file using its File ID (aka CNID or objID), or simply by using a NSURL object that keeps track of the file.

The secret to this working is that HFS(+) volumes assign each file and folder a unique number (the CNID), and so even if the file is renamed or moved, it can be looked up by that number again. Bookmarks also make use of this number by storing it for this purpose.

This also means that if you place your file on a file system that does not support file IDs, e.g. on a SMB network volume or a FAT formatted disk, renaming and moving of such files may NOT be detected by those apps, because there's no alternative tracking mechanism in this case.

See also Apple's Technical QA1113: The /.vol/ dir and volFS.

APFS, Apple's new volume format, does support File IDs as well, BTW.