Compression and In-Situ Query Processing for Fine-Grained Array Lineage
<|reference_start|>Compression and In-Situ Query Processing for Fine-Grained Array Lineage: Tracking data lineage is important for data integrity, reproducibility, and debugging data science workflows. However, fine-grained lineage (i.e., at a cell level) is challenging to store, even for the smallest datasets. This paper introduces DSLog, a storage system that efficiently stores, indexes, and queries array data lineage, agnostic to capture methodology. A main contribution is our new compression algorithm, named ProvRC, that compresses captured lineage relationships. Using ProvRC for lineage compression result in a significant storage reduction over functions with simple spatial regularity, beating alternative columnar-store baselines by up to 2000x}. We also show that ProvRC facilitates in-situ query processing that allows forward and backward lineage queries without decompression - in the optimal case, surpassing baselines by 20x in query latency on random numpy pipelines.<|reference_end|>
@article{zhao2024compression,
    title={Compression and In-Situ Query Processing for Fine-Grained Array Lineage},
    author={Jinjin Zhao, Sanjay Krishnan},
    journal={arXiv preprint arXiv:2405.17701},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17701},
    primaryClass={cs.DB}
}