Moving Hadoop to the Cloud: Harnessing Cloud Features and Flexibility for Hadoop Clusters by Bill Havanki

Author:Bill Havanki [Havanki, Bill]
Language: eng
Format: azw3
Publisher: O'Reilly Media
Published: 2017-07-14T04:00:00+00:00

$ sudo cp /opt/spark/conf/spark-env.sh.template /opt/spark/conf/spark-env.sh

Then, define the HADOOP_CONF_DIR environment variable, pointing to /etc/hadoop, so that Spark can find the cluster’s configuration files. If you installed the Spark package for user-provided Hadoop, then also define the SPARK_DIST_CLASSPATH environment variable so that Spark can locate Hadoop binaries:

export HADOOP_CONF_DIR=/etc/hadoop export SPARK_DIST_CLASSPATH=$(/opt/hadoop/bin/hadoop classpath)

This is a good checkpoint to make sure that Spark is configured correctly, before continuing to connect it to YARN. Try running the “SparkPi” example job locally:

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.