# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode # - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf) export SPARK_LOG_DIR=/tmp/spark-ss # - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp) # - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER) # - SPARK_NICENESS The scheduling priority for daemons. (Default: 0) Create different python environments Run list’ to list all installed packages and their version # ‘conda Other packages essential are : conda install pandas conda install matplotlib conda install psycopg2 (This package is to connect to postgreSQL database) In order to install any packages behind the proxy, we need to create a hidden file in user’s ‘.condarc’ folder (C:\Users\f401569). And the file is a plain text file containing the following: # This is a sample .condarc file. It adds the r Anaconda.org channel and enables # the show_channel_urls option. # channel locations. override conda defaults, i.e., conda will # search *only* the channels listed here, in the order given. Use "defaults" to # automatically include all default channels. Non-url channels will be # interpreted as Anaconda.org usernames (this can be changed by modifying the # channel_alias key; see below). The default is just 'defaults'. channels: - r - defaults # Show channel URLs when displaying what is going to be downloaded and # in list'. The default is False. show_channel_urls: true # See for more information about this file
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode # - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf) export SPARK_LOG_DIR=/tmp/spark-ss # - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp) # - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER) # - SPARK_NICENESS The scheduling priority for daemons. (Default: 0) Create different python environments Run ‘conda list’ to list all installed packages and their version # Other packages essential are : conda install pandas conda install matplotlib conda install psycopg2 (This package is to connect to postgreSQL database) In order to install any packages behind the proxy, we need to create a hidden file ‘.condarc’ in user’s folder (C:\Users\f401569). And the file is a plain text file containing the following: # This is a sample .condarc file. It adds the r Anaconda.org channel and enables # the show_channel_urls option. # channel locations. override conda defaults, i.e., conda will # search *only* the channels listed here, in the order given. Use "defaults" to # automatically include all default channels. Non-url channels will be # interpreted as Anaconda.org usernames (this can be changed by modifying the # channel_alias key; see below). The default is just 'defaults'. channels: - r - defaults # Show channel URLs when displaying what is going to be downloaded and # in list'. The default is False. show_channel_urls: true # See for more information about this file
SparkPythonUnix-Code-Examples_2.docx