Machine Learning, Dask, Python, Data Science, Linear Models.
the overhead involved with getting data off of a disk — this is the case whether we’re running Dask in local mode or on a large cluster. Typically, we’d want each chunk to be somewhere between 10MB and 1GB to minimize the IO overhead. I’ve selected a chunk size of 5000 rows per chunk in this