# ERR\_SPARK\_FAILED\_YARN\_KILLED\_MEMORY: Spark failure: killed by YARN (excessive memory usage)[¶](https://doc.dataiku.com/dss/latest/troubleshooting/errors/ERR_SPARK_FAILED_YARN_KILLED_MEMORY.html#err-spark-failed-yarn-killed-memory-spark-failure-killed-by-yarn-excessive-memory-usage "Permalink to this headline")

## Description[¶](https://doc.dataiku.com/dss/latest/troubleshooting/errors/ERR_SPARK_FAILED_YARN_KILLED_MEMORY.html#description "Permalink to this headline")

This error can happen when running any Spark-enabled recipe, when Spark is running in “YARN” deployment mode.

This error indicates that YARN (the resource manager) has forcefully killed the Spark components, because they ran above their allocated memory allocation.

When a Spark application starts on YARN, it tells YARN how much memory it will use at maximum. YARN accordingly reserves this amount of memory. If, during runtime, the memory usage (per container) goes above this limit, YARN kills the process for breaching its promise.

## Remediation[¶](https://doc.dataiku.com/dss/latest/troubleshooting/errors/ERR_SPARK_FAILED_YARN_KILLED_MEMORY.html#remediation "Permalink to this headline")

When a Spark application runs on YARN, it requests YARN containers with an amount of memory computed as: `spark.executor.memory + spark.yarn.executor.memoryOverhead`

`spark.executor.memory` is the amount of Java memory (Xmx) that Spark executors will get. However, Java processes always consume a bit more memory, which is accounted for by `spark.yarn.executor.memoryOverhead`

By default, the memory overhead is 10% of executor memory (with a minimum of 384 MB). This value is often not enough.

The remediation is to increase the value of `spark.yarn.executor.memoryOverhead` Spark setting. For more information about how to set Spark settings, please see Spark configurations. Note that your administrator may need to perform this change.

Beware: unlike `spark.executor.memory` where values like `3g` are permitted, the value for `spark.yarn.executor.memoryOverhead` must always be an integer, in megabytes.

We generally recommend setting a value between `500` and `1000`.
