Metadata can generally be referred to as data that describes other data. More specifically, filesystem metadata may include descriptive information about the context, quality, condition, and characteristics of data within a filesystem. A filesystem typically includes a hierarchy of directories (or a directory tree) that is used to organize files on a computer. For example, within some filesystems, an item of metadata may describe an individual data point or content item, or an item of metadata may describe a file or collection of data, wherein the file or collection of data may include a hierarchical structure which defines the relationship between different data points within the file. In particular, metadata may include information such as the name or title, size, length, ownership, location, etc. for any data point or collection of data.
Within many filesystems, the metadata that describes the various files and/or directories in the filesystem is stored in a data structure or a series of data structures called inodes. Each inode, which is assigned a unique inode number within that filesystem, is designed to store up to a certain amount of information or metadata about the file. Further, an “inode chunk” is a collection of a certain number of inodes stored contiguously on disk. Additionally, the size of the inodes is typically set when the filesystem is created.
Over time, new features may be implemented in the filesystem software. These features may require changes or additions to the metadata contained in the filesystem's inodes. Eventually, as more additions or changes are made, there may be no space left in the inode to support desired new functionality. Therefore, in such situations, there is a need to convert metadata contained in the inodes from the existing format to another format.
Previous attempts to convert metadata have resulted in the process being unreasonably time consuming, thereby causing excessive downtime of the filesystem, or, more importantly, an inadequate uptime of the filesystem. Additionally, the integrity of the metadata may be at risk due to the possibility of failure during the conversion process.
When converting a filesystem with hundreds of gigabytes of metadata, efficiency and reliability are of the upmost importance. Filesystems can contain hundreds of millions of inodes consuming hundreds of gigabytes of disk space. Metadata storage can typically perform at most a few hundred input/output (I/O) operations per second, so the time to rapidly convert a large filesystem can become unreasonable. Since metadata objects tend to be relatively small and generally not contiguous on disk, an efficient scheme for converting metadata cannot rely on reading items one by one from disk and chasing associating structures.
Another aspect of efficiency is the amount of storage required. The conversion process should not require disproportionate amounts of new storage nor render existing storage unusable due to fragmentation.
Clearly, reliability in any metadata conversion process is also important. A filesystem with hundreds of millions of files has great value to the user. Large filesystems take longer to convert, and the greater the conversion time, the greater the risk for failure during the conversion process. Accordingly, any viable scheme for converting metadata must allow for recovery from any failures, for example, process and/or node failures, during the conversion.