Spark Values

The following settings are added to the Spark submission in alpine.conf, where you can edit them if you choose. While these can be used to define some overall settings, any Spark tuning you define at the operator level takes precedence.

Name Default value Notes
spark.yarn.api.timeout 5000
spark.yarn.report.interval 2000
spark.executor.extraJavaOpts
spark.rdd.compress true
spark.io.compression.codec "org.apache.spark.io.SnappyCompressionCodec"
spark.eventLog.enabled false
spark.dynamicAllocation.enabled true Dynamic allocation can be disabled at the operator level. Regardless of this value, we use only dynamic allocation if your cluster is correctly configured to enable dynamic allocation.
spark.driver.maxResultSize 2g A Spark value that sets the maximum size of data that can be brought back to the driver.
spark.driver.extraJavaOptions -XX:MaxPermSize=256m