CheatSheet
  • Introduction
  • Garbage Collection
  • Scala
  • Kafka
  • Shell Scripting
  • Notebook
  • Hadoop
  • Git
  • Mac Terminal
  • IntelliJ Idea
  • Garbage Collection
  • YARN
  • NoSQL
  • DBConnection
  • ProjectBuild
  • Microservices
  • WebServices
  • Cloud Services
  • iTerm2
  • SBT
  • Useful Java Libraries
  • Python
  • Postgres
  • UML Class Diagram
  • Groovy
  • Mac Brew Install
  • Maven
  • Rough Works
  • Spark
  • AWS
  • Kubernetes
  • Docker
  • Redis
  • Telnet
Powered by GitBook
On this page

Was this helpful?

Spark

Tuning Apache Spark

  1. Dynamic Executor Allocation

    • Better Resource utilization

    • Good for multi-tenant environment

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.executpeIdleTimeout=2m
spark.dynamicAllocation.minExecutors=1
spark.dynamicAllocation.maxExecutors=2000

spark application information can be seen from Spark Context WebUI

Client Mode -> first application statrts at port 4040

PreviousRough WorksNextAWS

Last updated 2 years ago

Was this helpful?