
Learning Apache Spark 2
By :

When you set up Spark, executors are run on the nodes in the cluster. To put it simply, executors are the processes where you:
Each application has its own executor processes and they will stay up and running until your application is up and running. So by definition, they seem to be quite important from a performance perspective, and hence the three key metrics during a Spark deployment are:
So how do you allocate physical resources to Spark? While this may generally depend on the nature of the workload, you can vary between the following extreme parameters.
Figure 11.2: Executor granularity
Extreme approaches are generally a bad option except for very specific workload situations. For example, if you define very small sized executors...