diff --git a/README.md b/README.md index 724eeae..43e492a 100644 --- a/README.md +++ b/README.md @@ -151,4 +151,20 @@ SparkSession spark = SparkSession 4.打开spark server界面,可以看到已经完成的spark作业。 ![](https://raw.githubusercontent.com/huzekang/picbed/master/20190626112849.png) -### 提交作业到yarn \ No newline at end of file +### 提交作业到yarn +1.代码中定义的上下文不要指定master +```java + SparkSession spark = SparkSession + .builder() + .appName("Java Spark SQL Starter !!") + .enableHiveSupport() + .config("spark.some.config.option", "some-value") + .getOrCreate(); +``` + +2.使用`mvn clean package`打包好的作业,并提交到本地安装好的spark环境上跑 +``` +~/opt/spark-2.4.0-bin-hadoop2.7 » bin/spark-submit --master yarn --deploy-mode cluster --class "com.wugui.sparkstarter.SparkHiveNewVersion" /Users/huzekang/study/spark-starter/target/spark-starter-1.0-SNAPSHOT.jar +``` +3.打开yarn观察到作业已经完成了。 +![](https://raw.githubusercontent.com/huzekang/picbed/master/20190626133707.png) \ No newline at end of file diff --git a/src/main/java/com/wugui/sparkstarter/SparkHiveNewVersion.java b/src/main/java/com/wugui/sparkstarter/SparkHiveNewVersion.java index 0fdf924..8c6b36c 100644 --- a/src/main/java/com/wugui/sparkstarter/SparkHiveNewVersion.java +++ b/src/main/java/com/wugui/sparkstarter/SparkHiveNewVersion.java @@ -15,9 +15,10 @@ public static void main(String[] args) { // 定义上下文 SparkSession spark = SparkSession .builder() - // 如果需要提交到remote spark则使用spark://host:port + // 如果需要作业要以jar包形式提交到remote spark,则使用spark://host:port // .master("spark://10.0.0.50:7077") - // 如果需要提交到remote spark则使用local + // 如果idea中测试则使用local。 + // 如果作业要以jar包形式提交到yarn则不设置master。 .master("local") .appName("Java Spark SQL Starter !!") .enableHiveSupport()