YARN Configuration Values

To read information about your cluster, we access the YARN API connected to it. If we cannot contact the YARN API (usually due to security or ports being blocked), we use the following default values for your cluster size.

Name Notes
assumed.yarn.number.nodes = "4" Number of nodes (and thus how many containers) the cluster has available.
assumed.yarn.scheduler.maximum.allocation.vcores = "15" Assumed number of virtual cores in the YARN configuration file. If the YARN can be reached, we use yarn.scheduler.maximum.allocation.vcores. Otherwise, we use the "assumed" value.

If we cannot get the total available virtual cores for the cluster, we assume that it is the number of nodes multiplied by the cores per container, and that the container is the size of a node.

assumed.yarn.scheduler.maximum.allocation.mb= "8000" Assumed memory in megabytes available to each YARN container. We use the yarn.scheduler.maximum.allocation.mb value from the YARN configuration file if we can get it; otherwise, we use the "assumed" value.

This is the value used to cap the driver and executor memory. If we cannot determine the resources available on the cluster, we assume that the memory available on the cluster is the number of nodes multiplied by this value.

Related reference