Packages defined
A Quilt data package is a tree of serialized data wrapped in a software is a package instance. Package instances are immutable.
Leaf nodes in the package tree are called fragments or objects. Installed fragments are de-duplicated and kept in a local object store.
Package lifecycle
Quilt's core commands are build, push, and install. To use a data package you import it.
build creates a package
- Quilt uses pandas to parse tabular file formats into data frames and pyarrow to serialize data frames to Parquet format.
push stores a package in a server-side registry
- Packages are registered against a Flask/MySQL endpoint that controls permissions and keeps track of where data lives in blob storage (S3 for the Free tier).
install downloads a package
- After a permissions check the client receives a signed URL to download the package from blob storage.
- Packages are installed in the current directory in folder named
quilt_modules
import exposes your package to code
- Quilt data packages are wrapped in a Python module so that users can import data like code:
from quilt.data.USER_NAME import PACKAGE_NAME.
- Data
importis lazy to minimize I/O. Data are only loaded from disk if and when the user references the data directly.
- Quilt looks for packages in the current directory followed by all ancestors, in ascending order.
Diagram
