1. spark(기존 편집)
SPARK_HOME => /dahy/spark3
PYSPARK_PYTHON => python3
PYSPARK_DRIVER_PYTHON => python3
2. spark_stand (신규생성)
Interpreter Name => spark_stand
Interpreter group => spark
SPARK_HOME => /dahy/spark3
spark.master => spark://spark-master-01:7177
spark.submit.deployMode => client
spark.app.name => Zeppelin_STAND
spark.driver.cores => 1
spark.driver.memory => 1g
spark.executor.cores => 4
spark.executor.memory => 4g
spark.executor.instances => (*공백, 비워두기)
PYSPARK_PYTHON => python3
PYSPARK_DRIVER_PYTHON => python3
spark.cores.max => 10 (*신규 추가)
- standalone 실행 : $ /dahy/spark3/sbin/start-all.sh
3. spark_yarn
Interpreter Name => spark_yarn
Interpreter group => spark
SPARK_HOME => /dahy/spark3
spark.master => yarn
spark.submit.deployMode => client
spark.app.name => Zeppelin_YARN
spark.driver.cores => 1
spark.driver.memory => 1g
spark.executor.cores => 1
spark.executor.memory => 2g
spark.executor.instances => 5
PYSPARK_PYTHON => python3
PYSPARK_DRIVER_PYTHON => python3
YARN_CONF_DIR => /dahy/spark3/conf2 (*신규 추가)
- yarn 실행 : $ /dahy/hadoop3/sbin/start-all.sh
'data processing > spark' 카테고리의 다른 글
Spark RDD 프로그래밍 (0) | 2023.01.24 |
---|---|
spark 소개 (0) | 2023.01.24 |
Zeppelin 환경구성 (0) | 2022.12.29 |
SPARK 서버 환경 구성 (0) | 2022.12.28 |
댓글