Launching Spark on YARN

Launching Spark on YARN

Launching YARN

[hadoop@testmain ~]$ cd $HADOOP_HOME
[hadoop@testmain hadoop]$ ./sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-testmain.out
localhost: starting nodemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-testmain.out

訪問8080端口，確認YARN正常運作

Configuration

[hadoop@testmain hadoop]$ cd $SPARK_HOME
[hadoop@testmain spark]$ vim conf/spark-env.sh
## 將HADOOP_CONF_DIR設定至Hadoop的config文件所在資料夾
HADOOP_CONF_DIR="/opt/software/hadoop/etc/hadoop"

Launching Spark on YARN

[hadoop@testmain spark]$ ./bin/spark-submit \
> --jars /opt/software/hive/lib/mysql-connector-java-5.1.44-bin.jar \
> --master yarn \
> --class org.apache.spark.examples.SparkPi \
> /opt/software/spark/examples/jars/spark-examples*.jar \
> 3
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/software/alluxioBuild1.6/client/spark/alluxio-1.6.0-spark-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/software/spark-2.2.0-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/03/29 00:49:01 INFO spark.SparkContext: Running Spark version 2.2.0
18/03/29 00:49:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/29 00:49:02 INFO spark.SparkContext: Submitted application: Spark Pi
18/03/29 00:49:02 INFO spark.SecurityManager: Changing view acls to: hadoop
18/03/29 00:49:02 INFO spark.SecurityManager: Changing modify acls to: hadoop
18/03/29 00:49:02 INFO spark.SecurityManager: Changing view acls groups to: 
18/03/29 00:49:02 INFO spark.SecurityManager: Changing modify acls groups to: 
18/03/29 00:49:02 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/03/29 00:49:02 INFO util.Utils: Successfully started service 'sparkDriver' on port 34636.
18/03/29 00:49:02 INFO spark.SparkEnv: Registering MapOutputTracker
18/03/29 00:49:02 INFO spark.SparkEnv: Registering BlockManagerMaster
18/03/29 00:49:02 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/03/29 00:49:02 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/03/29 00:49:02 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-d1e1b957-5a1a-4ebe-a4cd-ae7b3cfa8298
18/03/29 00:49:02 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
18/03/29 00:49:02 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/03/29 00:49:02 INFO util.log: Logging initialized @2176ms
18/03/29 00:49:02 INFO server.Server: jetty-9.3.z-SNAPSHOT
18/03/29 00:49:02 INFO server.Server: Started @2285ms
18/03/29 00:49:02 INFO server.AbstractConnector: Started ServerConnector@750a562{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/03/29 00:49:02 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
## ...
18/03/29 00:49:03 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.128.91:4040
18/03/29 00:49:03 INFO spark.SparkContext: Added JAR file:/opt/software/spark/examples/jars/spark-examples_2.11-2.2.0.jar at spark://192.168.128.91:34636/jars/spark-examples_2.11-2.2.0.jar with timestamp 1522255743082
18/03/29 00:49:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/03/29 00:49:04 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
18/03/29 00:49:04 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
18/03/29 00:49:04 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
18/03/29 00:49:04 INFO yarn.Client: Setting up container launch context for our AM
18/03/29 00:49:04 INFO yarn.Client: Setting up the launch environment for our AM container
18/03/29 00:49:04 INFO yarn.Client: Preparing resources for our AM container
18/03/29 00:49:06 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/03/29 00:49:08 INFO yarn.Client: Uploading resource file:/tmp/spark-d9c4bb6e-260d-48fc-956c-c4aaa6062307/__spark_libs__3434199402497597155.zip -> hdfs://192.168.128.91:9000/user/hadoop/.sparkStaging/application_1522255678806_0001/__spark_libs__3434199402497597155.zip
18/03/29 00:49:10 INFO yarn.Client: Uploading resource file:/opt/software/hive/lib/mysql-connector-java-5.1.44-bin.jar -> hdfs://192.168.128.91:9000/user/hadoop/.sparkStaging/application_1522255678806_0001/mysql-connector-java-5.1.44-bin.jar
18/03/29 00:49:10 INFO yarn.Client: Uploading resource file:/tmp/spark-d9c4bb6e-260d-48fc-956c-c4aaa6062307/__spark_conf__4395531597464213123.zip -> hdfs://192.168.128.91:9000/user/hadoop/.sparkStaging/application_1522255678806_0001/__spark_conf__.zip
## ...
18/03/29 00:49:10 INFO yarn.Client: Submitting application application_1522255678806_0001 to ResourceManager
18/03/29 00:49:10 INFO impl.YarnClientImpl: Submitted application application_1522255678806_0001
18/03/29 00:49:10 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1522255678806_0001 and attemptId None
18/03/29 00:49:11 INFO yarn.Client: Application report for application_1522255678806_0001 (state: ACCEPTED)
18/03/29 00:49:11 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: [Thu Mar 29 00:49:11 +0800 2018] Scheduler has assigned a container for AM, waiting for AM container to be launched
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1522255750717
	 final status: UNDEFINED
	 tracking URL: http://testmain:8088/proxy/application_1522255678806_0001/
	 user: hadoop
18/03/29 00:49:12 INFO yarn.Client: Application report for application_1522255678806_0001 (state: ACCEPTED)
## ...
18/03/29 00:49:31 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 2.702 s
18/03/29 00:49:31 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 3.037696 s
Pi is roughly 3.1408904696348987
18/03/29 00:49:31 INFO server.AbstractConnector: Stopped Spark@750a562{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/03/29 00:49:31 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.128.91:4040
18/03/29 00:49:31 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
18/03/29 00:49:31 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
18/03/29 00:49:31 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
18/03/29 00:49:31 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
18/03/29 00:49:31 INFO cluster.YarnClientSchedulerBackend: Stopped
18/03/29 00:49:31 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/03/29 00:49:31 INFO memory.MemoryStore: MemoryStore cleared
18/03/29 00:49:31 INFO storage.BlockManager: BlockManager stopped
18/03/29 00:49:31 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
18/03/29 00:49:31 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/03/29 00:49:31 INFO spark.SparkContext: Successfully stopped SparkContext
18/03/29 00:49:31 INFO util.ShutdownHookManager: Shutdown hook called
18/03/29 00:49:31 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-d9c4bb6e-260d-48fc-956c-c4aaa6062307

Log Explanation

ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.128.91:4040
- 啟動SparkUI在4040端口上
spark.SparkContext: Added JAR file:/opt/software/spark/examples/jars/spark-examples_2.11-2.2.0.jar at spark://192.168.128.91:34636/jars/spark-examples_2.11-2.2.0.jar with timestamp 1522255743082
- 開始上傳指定的jar包
client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
- 連接到在8032端口上的ResourceManager
yarn.Client: Requesting a new application from cluster with 1 NodeManagers
- 請求一個NodeManager，讓Application master使用
Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
- 檢查Application沒有要求超過cluster的最大記憶體容量
yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
- 將會為Application master container配置896 MB的記憶體
yarn.Client: Setting up container launch context for our AM
- 建立Application master container
yarn.Client: Setting up the launch environment for our AM container
- 建立Application master container的運行環境
yarn.Client: Preparing resources for our AM container
- 準備Application master container的資源
yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
- 此為調優點，要將SPARK_HOME/jars下的jar包上傳至HDFS上，減少libraries上傳造成的網路負擔
yarn.Client: Submitting application application_1522255678806_0001 to ResourceManager
- 提交一個Application給ResourceManager，id為application_1522255678806_0001