Dask
Overview
Dask is a package focused on interacting with data in Python. Dask has a lot of similar functionality to a common Python package named Pandas. The advantage of Dask is that it is a naturally parallel package. This means that the people who wrote Dask utilized parallel processing in order to help it read and work with data faster than other packages, like Pandas, that use a single processer.
Dask Dashboard
In addition to the built-in parallel processing Dask has a dashboard client that can be used to monitor the processes as they run. Because the Data Mine code is run on Purdue’s high performance computing cluster we have a few extra steps to allow us to view the dashboard.
Steps to Launch a Dask Dashboard
Navigate to the Purdue HPC website and click the
Remote Desktopoption.
Note: this will require the dual-authentication for Purdue’s systems.
Launch a terminal in the
Remote Desktopsession. This can be done by either:
Selecting the terminal icon that appears in the bottom center of the screen.
Right-clicking on the desktop and selecting the
Open Terminal Hereoption.
Once we have a terminal open we need to load the Jupyter Lab environment so that we can launch a new session. The commands to load this environment and launch a jupyter lab session are included below:
In the terminal type
module use /scratch/brown/kamstut/tdm/opt/modulefiles
Once we have run the first command type
module load python/jupyterlab-py3.9.6
These two commands are specific to the Data Mine environment. The first command makes specific modules that the staff created available to the user. The second loads the Jupyter Lab environment that we will use for the Dask dashboard.
After the module has been loaded run the
python -m Jupyter Labcommand in the terminal. This will launch a new Jupyter Lab session and should open the browser automatically.
In the Jupyter Lab session you can now launch the Dask dashboard. The code for the dashboard is included below:
from dask.distributed import Client client = Client() client
Once the code cell above is exectued Dask should provide a link to the dashboard. Clicking the link will open the dashboard in a new browser tab.