Machine Learning, Dask, Python, Data Science, Linear Models.
noticed after the concatenation that the array had a chunk size of (1,100). This means that each chunk contains one row and 100 columns. The ZARR format writes one file per chunk, meaning we’d produce 568,454 individual files if we didn’t rechunk the data. This is extremely inefficient because of