Rethinking Files
I&#x27;ve done some silly things [0] with python&#x27;s pathlib recently that seem related to the issues discussed here. Given that smalltalk message passing finally clicked for me durnig the process, I am attracted to an object-like solution for everything (or a file-object-like solution for everything, since the practical performance advantages are undeniable). That said there are some considerations both for the low level implementation, and for high level things like affordances for &#x27;file&#x27; operations.<p>In direct response to the suggestion about file paths for verbs. Allan Kay says in one (possibly many) of his talks something along the lines of &#x27;every function should have a url.&#x27; The one of surely many challenges is how to ensure that the mechanism used to populate file system paths with nested functionality (e.g. &#x2F;usr&#x2F;bin&#x2F;ls&#x2F;all to `ls -a`) don&#x27;t trigger malicious behavior during service&#x2F;capability discovery. Being able to more deeply introspect file data and metadata as if the file were a folder could potentially be implemented as a plugin, and I worry about the complexity of requiring a file system to know about the contents of the files that it hosts, or that the files themselves be required to know about how to tell the file system about themselves. Existing file systems adhere to a fairly strict separation of concerns, since who knows what new file format or language will appear, and who knows what file system the file will need to exist on.<p>Said another way I think that the primary issue with the suggested approach is that it is hard to extend. The file system itself needs to know about the new type of object that it is going to represent, rather than simply acting as an index of paths to all objects. If there is a type of object that is opaque to the current version of the file system that object either has to implement a file-system-specific discovery protocol (which surely would have fun security considerations if it were anything other than a static manifest) or the user has to wait for a new version of the file system that knows what to do with that file type.<p>Some thoughts from my own work. (partially in the context of OJFord&#x27;s comment below)<p>Treating files and urls as objects that have identifiers, metadata, and data portions and where the data portion is treated as a generator is very powerful, but the affordances around the expression local_file.data = remote_file.data make me hesitate. When assignment can trigger a network saturating read operation, or when setter doesn&#x27;t know anything about how much space is on a disk, etc. then there are significant footguns waiting to be fired and I have already shot myself a couple of times.<p>The more homogeneous the object interface the better. However, this comes with a major risk. If the underlying systems you are wrapping have different operational semantics (think files system vs database transactions) and there is no way to distinguish between them based solely on the interface (because it is homogeneous) then disaster will strike at some point due to a mismatch. To avoid this everything built on top of the object representation has to be implemented under the assumption of the worst case possible behavior, making it difficult to leverage the features of more advanced systems. As with the affordances around local.data = remote.data, if I have a socket, a local file, a remote web page that I own, a handle to an led, a handle to a stop light, a database row in a table that has triggers set, the stdin to an open ssh session, and a network ring buffer all represented in the same object system, I have as many meanings for file_object.write(&#x27;something&#x27;) as I have types of objects, and the consequences and side effects of calling write are so diverse (from flipping bits on a harddrive to triggering arbitrary code execution) that it is all but guaranteed that something will go horribly wrong. At the very least there would need to be a distinction between operations where all side effects could be accounted for beforehand (e.g. writing a file of known length to disk has the side effect of reducing free disk space, but that is known before the operation starts), and operations where the consequences will depend on the contents of the message (e.g. DROP TABLES), with perhaps a middle ground for cases with static side effects (e.g. the database trigger) but that would not immediately visible to the caller and that might change from time to time.<p>The distinction between files and folders is quite annoying (non-homogeneous), especially if you want to require that certain pieces of metadata always &#x27;follow&#x27; a file. This is from working with xattrs that are extremely easy to loose if you aren&#x27;t careful. Xattrs are a great engineering optimization to make use of dead space in the file system data structure, but they aren&#x27;t quite the full abstraction one would want. It is also not entirely clear what patterns to use when you have a file that is also a folder -- do you make the metadata the outer file and the data the inner file? Or the other way around? Having the metadata as the outer file means that you can change the metdata without changing the data, but that the metadata will always &#x27;update&#x27; when its contents (the data) changes. However, when I first thought about using such a system, I had it the other way around, and a system with that much flexibility I suspect would have even more footguns than the current system.<p>Another issue is the long standing question around what constitutes an atomic operation. Everything is simple if only a single well behaved program is ever going to touch the files, but trying to build a full object-like system on top of existing systems is a recipe for leaky abstraction nightmares.<p>While I was working on this I came across debates from before I was born. For example hardlinks vs symlinks. There are real practical engineering tradeoffs that I can&#x27;t even begin to comment on because I don&#x27;t understand the use cases for hardlinks well enough to say why we didn&#x27;t just get rid of them entirely.<p>0. <a href="https:&#x2F;&#x2F;github.com&#x2F;SciCrunch&#x2F;sparc-curation&#x2F;blob&#x2F;master&#x2F;sparcur&#x2F;paths.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;SciCrunch&#x2F;sparc-curation&#x2F;blob&#x2F;master&#x2F;spar...</a>