Zeppelin(spark) 인터프리터 설정

1. spark(기존 편집)

SPARK_HOME => /dahy/spark3
PYSPARK_PYTHON => python3
PYSPARK_DRIVER_PYTHON => python3

 

2. spark_stand (신규생성)

Interpreter Name => spark_stand
Interpreter group => spark
SPARK_HOME => /dahy/spark3
spark.master => spark://spark-master-01:7177
spark.submit.deployMode => client
spark.app.name => Zeppelin_STAND
spark.driver.cores => 1
spark.driver.memory => 1g
spark.executor.cores => 4
spark.executor.memory => 4g
spark.executor.instances => (*공백, 비워두기)
PYSPARK_PYTHON => python3
PYSPARK_DRIVER_PYTHON => python3
spark.cores.max => 10 (*신규 추가)

- standalone 실행 : $ /dahy/spark3/sbin/start-all.sh

 

3. spark_yarn

Interpreter Name => spark_yarn
Interpreter group => spark
SPARK_HOME => /dahy/spark3
spark.master => yarn
spark.submit.deployMode => client
spark.app.name => Zeppelin_YARN
spark.driver.cores => 1
spark.driver.memory => 1g
spark.executor.cores => 1
spark.executor.memory => 2g
spark.executor.instances => 5
PYSPARK_PYTHON => python3
PYSPARK_DRIVER_PYTHON => python3
YARN_CONF_DIR => /dahy/spark3/conf2 (*신규 추가)

- yarn 실행 : $ /dahy/hadoop3/sbin/start-all.sh