YARN Configuration Values
To read information about your cluster, we access the YARN API connected to it. If we cannot contact the YARN API (usually due to security or ports being blocked), we use the following default values for your cluster size.
Name | Notes |
---|---|
assumed.yarn.number.nodes = "4" | Number of nodes (and thus how many containers) the cluster has available. |
assumed.yarn.scheduler.maximum.allocation.vcores = "15" | Assumed number of virtual cores in the YARN configuration file. If the YARN can be reached, we use
yarn.scheduler.maximum.allocation.vcores. Otherwise, we use the "assumed" value.
If we cannot get the total available virtual cores for the cluster, we assume that it is the number of nodes multiplied by the cores per container, and that the container is the size of a node. |
assumed.yarn.scheduler.maximum.allocation.mb= "8000" | Assumed memory in megabytes available to each YARN container. We use the
yarn.scheduler.maximum.allocation.mb value from the YARN configuration file if we can get it; otherwise, we use the "assumed" value.
This is the value used to cap the driver and executor memory. If we cannot determine the resources available on the cluster, we assume that the memory available on the cluster is the number of nodes multiplied by this value. |
Copyright © Cloud Software Group, Inc. All rights reserved.