yarn 管理内存相关配置
| properties | value | 描述 | description |
|---|---|---|---|
| yarn.nodemanager.resource.memory-mb | -1 | 可分配给containers的物理内存量（以MB为单位）。 如果设置为-1并且yarn.nodemanager.resource.detect-hardware-capabilities为true，则将自动计算该值（在Windows和Linux中）。 在其他情况下，默认值为8192MB。 | Amount of physical memory, in MB, that can be allocated for containers. If set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically calculated(in case of Windows and Linux). In other cases, the default is 8192MB. |
| yarn.nodemanager.resource.detect-hardware-capabilities | false | 启用对节点容量（例如内存和CPU）的自动检测功能。 | Enable auto-detection of node capabilities such as memory and CPU. |
| yarn.nodemanager.resource.system-reserved-memory-mb | -1 | 为非YARN进程保留的物理内存量（以MB为单位）。 仅当yarn.nodemanager.resource.detect-hardware-capabilities设置为true且yarn.nodemanager.resource.memory-mb为-1时，才使用此配置。 如果设置为-1，则此值计算为（系统内存-2 * HADOOP_HEAPSIZE）的20％ | Amount of physical memory, in MB, that is reserved for non-YARN processes. This configuration is only used if yarn.nodemanager.resource.detect-hardware-capabilities is set to true and yarn.nodemanager.resource.memory-mb is -1. If set to -1, this amount is calculated as 20% of (system memory – 2*HADOOP_HEAPSIZE) |
| yarn.nodemanager.vmem-pmem-ratio | 2.1 | 设置容器的内存限制时，虚拟内存与物理内存之间的比率。 容器分配以物理内存的形式表示，虚拟内存使用量可以超出此分配比例。 | Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio |
| yarn.nodemanager.pmem-check-enabled | true | 是否启动一个线程监测每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉 | Whether physical memory limits will be enforced for containers |
| yarn.nodemanager.vmem-check-enabled | true | 是否启动一个线程监测每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉 | Whether virtual memory limits will be enforced for containers. |
| yarn.scheduler.maximum-allocation-mb | 8192 | RM为每个container请求分配的最大内存（以MB为单位）。 高于此值的内存请求将引发InvalidResourceRequestException。 | The maximum allocation for every container request at the RM in MBs. Memory requests higher than this will throw an InvalidResourceRequestException. |
| yarn.scheduler.minimum-allocation-mb | 1024 | RM处每个container请求分配的最小内存（以MB为单位）。 低于此值的内存请求将设置为此属性的值。 此外，资源管理器将关闭配置为具有少于此值的内存的节点管理器。 | The minimum allocation for every container request at the RM in MBs. Memory requests lower than this will be set to the value of this property. Additionally, a node manager that is configured to have less memory than this value will be shut down by the resource manager. |
yarn 管理vcores相关配置
| properties | value | 描述 | description |
|---|---|---|---|
| yarn.nodemanager.resource.cpu-vcores | -1 | 可以为容器分配的vcore数。 RM调度程序在为容器分配资源时使用它。 这不是用来限制YARN容器使用的CPU数量。 如果将其设置为-1，并且yarn.nodemanager.resource.detect-hardware-capabilities为true，则在Windows和Linux情况下会自动从硬件确定它。 在其他情况下，默认情况下，vcore的数量为8。 | Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of CPUs used by YARN containers. If it is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically determined from the hardware in case of Windows and Linux. In other cases, number of vcores is 8 by default. |
| yarn.nodemanager.resource.pcores-vcores-multiplier | -1 | 确定如何将physcal cores转换为vcore的乘数。 如果将yarn.nodemanager.resource.cpu-vcores设置为-1（这意味着自动计算vcore），并且yarn.nodemanager.resource.detect-hardware-capabilities设置为true，则使用此值。 vcore的数量将计算为CPUs * multiplier. | Multiplier to determine how to convert phyiscal cores to vcores. This value is used if yarn.nodemanager.resource.cpu-vcores is set to -1(which implies auto-calculate vcores) and yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The number of vcores will be calculated as number of CPUs * multiplier. |
| yarn.scheduler.minimum-allocation-vcores | 1 | 就vcores而言，RM处每个container请求分配的最小vcores数量。 低于此值的请求将设置为此属性的值。 此外，资源管理器将关闭配置为具有比该值更少的虚拟核的节点管理器。 | The minimum allocation for every container request at the RM in terms of virtual CPU cores. Requests lower than this will be set to the value of this property. Additionally, a node manager that is configured to have fewer virtual cores than this value will be shut down by the resource manager. |
| yarn.scheduler.maximum-allocation-vcores | 4 | 就vcores而言，RM上每个container请求分配的最大vcores数量。 高于此值的请求将引发InvalidResourceRequestException | The maximum allocation for every container request at the RM in terms of virtual CPU cores. Requests higher than this will throw an InvalidResourceRequestException. |
相关链接：
yarn-site官方链接：
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
版权声明：本文为jiajane原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。