The following instance properties relate to bulk import, i.e. ingesting data using Spark jobs running on EMR or EKS.
Note that on EMR, the total resource allocation must align with the instance types used for the cluster. For the maximum memory usage, combine the memory and memory overhead properties, and compare against the maximum memory allocation for YARN in the Hadoop task configuration:
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html
As an example, if we use m7i.xlarge for executor instances, that has a maximum allocation of 54272 MiB, or 53 GiB. If we want 3 executors per instance, we can have 53 GiB / 3 = 18,090.666 MiB per executor. We can set the executor memory to 16 GiB, and the executor memory overhead to the remainder of that amount, which is 18,090 MiB - 16 GiB = 1,706 MiB, or 1.666 GiB. This is just above the default Spark memory overhead factor of 0.1, i.e. 16 GiB x 0.1 = 1.6 GiB.
Also see EMR best practices:
https://aws.github.io/aws-emr-best-practices/docs/bestpractices/Applications/Spark/best_practices/#bp-516----tune-driverexecutor-memory-cores-and-sparksqlshufflepartitions-to-fully-utilize-cluster-resources
| Property Name | Description | Default Value | Run CDK Deploy When Changed |
|---|---|---|---|
| sleeper.bulk.import.class.name | The class to use to perform the bulk import. The default value below uses Spark Dataframes. There is an alternative option that uses RDDs (sleeper.bulkimport.runner.rdd.BulkImportJobRDDDriver). | sleeper.bulkimport.runner.dataframelocalsort.BulkImportDataframeLocalSortDriver | false |
| sleeper.bulk.import.emr.spark.shuffle.mapStatus.compression.codec | The compression codec for map status results. Used to set spark.shuffle.mapStatus.compression.codec. Stops "Decompression error: Version not supported" errors - only a value of "lz4" has been tested. |
lz4 | true |
| sleeper.bulk.import.emr.spark.speculation | If true then speculative execution of tasks will be performed. Used to set spark.speculation. See https://spark.apache.org/docs/latest/configuration.html. |
false | true |
| sleeper.bulk.import.spark.speculation.quantile | Fraction of tasks which must be complete before speculation is enabled for a particular stage. Used to set spark.speculation.quantile. See https://spark.apache.org/docs/latest/configuration.html. |
0.75 | true |
| sleeper.bulk.import.starter.memory.mb | The amount of memory in MB for lambda functions that start bulk import jobs. | true | |
| sleeper.bulk.import.emr.spark.executor.memory | The amount of memory allocated to a Spark executor. Used to set spark.executor.memory. See https://spark.apache.org/docs/latest/configuration.html. |
16g | true |
| sleeper.bulk.import.emr.spark.driver.memory | The amount of memory allocated to the Spark driver. Used to set spark.driver.memory. See https://spark.apache.org/docs/latest/configuration.html. |
16g | true |
| sleeper.bulk.import.emr.spark.executor.instances | The number of executors. Used to set spark.executor.instances. See https://spark.apache.org/docs/latest/configuration.html. |
29 | true |
| sleeper.bulk.import.emr.spark.executor.memory.overhead | The memory overhead for an executor. Used to set spark.executor.memoryOverhead. See https://spark.apache.org/docs/latest/configuration.html. |
1706m | true |
| sleeper.bulk.import.emr.spark.driver.memory.overhead | The memory overhead for the driver. Used to set spark.driver.memoryOverhead. See https://spark.apache.org/docs/latest/configuration.html. |
1706m | true |
| sleeper.bulk.import.emr.spark.default.parallelism | The default parallelism for Spark job. Used to set spark.default.parallelism. See https://spark.apache.org/docs/latest/configuration.html. |
290 | true |
| sleeper.bulk.import.emr.spark.sql.shuffle.partitions | The number of partitions used in a Spark SQL/dataframe shuffle operation. Used to set spark.sql.shuffle.partitions. See https://spark.apache.org/docs/latest/configuration.html. |
290 | true |
| sleeper.bulk.import.emr.keypair.name | (Non-persistent or persistent EMR mode only) An EC2 keypair to use for the EC2 instances. Specifying this will allow you to SSH to the nodes in the cluster while it's running. NOTE: Updating this property will cause any existing EMR cluster to be destroyed and re-created. | true | |
| sleeper.bulk.import.emr.master.additional.security.group | (Non-persistent or persistent EMR mode only) Specifying this security group causes the group to be added to the EMR master's list of security groups. | true | |
| sleeper.bulk.import.emr.spark.executor.cores | (Non-persistent or persistent EMR mode only) The number of cores used by an executor. Used to set spark.executor.cores. See https://spark.apache.org/docs/latest/configuration.html. |
5 | true |
| sleeper.bulk.import.emr.spark.driver.cores | (Non-persistent or persistent EMR mode only) The number of cores used by the driver. Used to set spark.driver.cores. See https://spark.apache.org/docs/latest/configuration.html. |
5 | true |
| sleeper.bulk.import.emr.spark.network.timeout | (Non-persistent or persistent EMR mode only) The default timeout for network interactions in Spark. Used to set spark.network.timeout. See https://spark.apache.org/docs/latest/configuration.html. |
800s | true |
| sleeper.bulk.import.emr.spark.executor.heartbeat.interval | (Non-persistent or persistent EMR mode only) The interval between heartbeats from executors to the driver. Used to set spark.executor.heartbeatInterval. See https://spark.apache.org/docs/latest/configuration.html. |
60s | true |
| sleeper.bulk.import.emr.spark.dynamic.allocation.enabled | (Non-persistent or persistent EMR mode only) Whether Spark should use dynamic allocation to scale resources up and down. Used to set spark.dynamicAllocation.enabled. See https://spark.apache.org/docs/latest/configuration.html. |
false | true |
| sleeper.bulk.import.emr.spark.memory.fraction | (Non-persistent or persistent EMR mode only) The fraction of heap space used for execution and storage. Used to set spark.memory.fraction. See https://spark.apache.org/docs/latest/configuration.html. |
0.80 | true |
| sleeper.bulk.import.emr.spark.memory.storage.fraction | (Non-persistent or persistent EMR mode only) The amount of storage memory immune to eviction, expressed as a fraction of the heap space used for execution and storage. Used to set spark.memory.storageFraction. See https://spark.apache.org/docs/latest/configuration.html. |
0.30 | true |
| sleeper.bulk.import.emr.spark.executor.extra.java.options | (Non-persistent or persistent EMR mode only) JVM options passed to the executors. Used to set spark.executor.extraJavaOptions. See https://spark.apache.org/docs/latest/configuration.html. |
-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' | true |
| sleeper.bulk.import.emr.spark.driver.extra.java.options | (Non-persistent or persistent EMR mode only) JVM options passed to the driver. Used to set spark.driver.extraJavaOptions. See https://spark.apache.org/docs/latest/configuration.html. |
-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' | true |
| sleeper.bulk.import.emr.spark.yarn.scheduler.reporter.thread.max.failures | (Non-persistent or persistent EMR mode only) The maximum number of executor failures before YARN can fail the application. Used to set spark.yarn.scheduler.reporterThread.maxFailures. See https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/. |
5 | true |
| sleeper.bulk.import.emr.spark.storage.level | (Non-persistent or persistent EMR mode only) The storage to use for temporary caching. Used to set spark.storage.level. See https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/. |
MEMORY_AND_DISK_SER | true |
| sleeper.bulk.import.emr.spark.rdd.compress | (Non-persistent or persistent EMR mode only) Whether to compress serialized RDD partitions. Used to set spark.rdd.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | true |
| sleeper.bulk.import.emr.spark.shuffle.compress | (Non-persistent or persistent EMR mode only) Whether to compress map output files. Used to set spark.shuffle.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | true |
| sleeper.bulk.import.emr.spark.shuffle.spill.compress | (Non-persistent or persistent EMR mode only) Whether to compress data spilled during shuffles. Used to set spark.shuffle.spill.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | true |
| sleeper.bulk.import.emr.ebs.volume.size.gb | (Non-persistent or persistent EMR mode only) The size of the EBS volume in gibibytes (GiB). This can be a number from 10 to 1024. |
256 | true |
| sleeper.bulk.import.emr.ebs.volume.type | (Non-persistent or persistent EMR mode only) The type of the EBS volume. Valid values are 'gp2', 'gp3', 'io1', 'io2'. |
gp2 | true |
| sleeper.bulk.import.emr.ebs.volumes.per.instance | (Non-persistent or persistent EMR mode only) The number of EBS volumes per instance. This can be a number from 1 to 25. |
4 | true |
| sleeper.bulk.import.emr.ebs.encryption.key.arn | ARN of the KMS Key used to encrypt data at rest on the local file system in AWS EMR. See https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-encryption-enable.html#emr-encryption-create-keys. |
true | |
| sleeper.bulk.import.emr.serverless.architecture | The architecture for EMR Serverless to use. X86_64 or ARM64 (Coming soon) | X86_64 | true |
| sleeper.bulk.import.emr.serverless.release | The version of EMR Serverless to use. | emr-7.12.0 | true |
| sleeper.bulk.import.emr.serverless.autostart.enabled | Set to true to allow an EMR Serverless Application to start automatically when a job is submitted. | true | true |
| sleeper.bulk.import.emr.serverless.autostop.enabled | Set to true to allow an EMR Serverless Application to stop automatically when there are no jobs to process. Turning this off with pre-initialised capacity turned off is not recommended. |
true | true |
| sleeper.bulk.import.emr.serverless.autostop.timeout | The number of minutes of inactivity before EMR Serverless stops the application. | 15 | true |
| sleeper.bulk.import.emr.serverless.spark.executor.cores | The number of cores used by a Serverless executor. Used to set spark.executor.cores. See https://spark.apache.org/docs/latest/configuration.html. |
4 | false |
| sleeper.bulk.import.emr.serverless.spark.executor.memory | The amount of memory allocated to a Serverless executor. Used to set spark.executor.memory. See https://spark.apache.org/docs/latest/configuration.html. |
16G | false |
| sleeper.bulk.import.emr.serverless.spark.emr-serverless.executor.disk | The amount of storage allocated to a Serverless executor. See https://spark.apache.org/docs/latest/configuration.html. |
200G | false |
| sleeper.bulk.import.emr.serverless.spark.executor.instances | The number of executors to be used with Serverless. Used to set spark.executor.instances. See https://spark.apache.org/docs/latest/configuration.html. |
36 | false |
| sleeper.bulk.import.emr.serverless.spark.driver.cores | The number of cores used by the Serverless Spark driver. Used to set spark.driver.cores. See https://spark.apache.org/docs/latest/configuration.html. |
4 | false |
| sleeper.bulk.import.emr.serverless.spark.driver.memory | The amount of memory allocated to the Serverless Spark driver. Used to set spark.driver.memory. See https://spark.apache.org/docs/latest/configuration.html. |
16G | false |
| sleeper.bulk.import.emr.serverless.spark.executorEnv.JAVA_HOME | The path to JAVA_HOME to be used by the custom image for bulk import. | /usr/lib/jvm/jre-11 | false |
| sleeper.bulk.import.emr.serverless.spark.dynamic.allocation.enabled | Whether Spark should use dynamic allocation to scale resources up and down. Used to set spark.dynamicAllocation.enabled. See https://spark.apache.org/docs/latest/configuration.html. | false | false |
| sleeper.bulk.import.emr.serverless.spark.rdd.compress | Whether to compress serialized RDD partitions. Used to set spark.rdd.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | false |
| sleeper.bulk.import.emr.serverless.spark.shuffle.compress | Whether to compress map output files. Used to set spark.shuffle.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | false |
| sleeper.bulk.import.emr.serverless.spark.shuffle.spill.compress | Whether to compress data spilled during shuffles. Used to set spark.shuffle.spill.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | false |
| sleeper.bulk.import.emr.serverless.spark.default.parallelism | The default parallelism for Spark job. Used to set spark.default.parallelism. See https://spark.apache.org/docs/latest/configuration.html. |
288 | false |
| sleeper.bulk.import.emr.serverless.spark.sql.shuffle.partitions | The number of partitions used in a Spark SQL/dataframe shuffle operation. Used to set spark.sql.shuffle.partitions. See https://spark.apache.org/docs/latest/configuration.html. |
288 | false |
| sleeper.bulk.import.emr.serverless.spark.network.timeout | The default timeout for network interactions in Spark. Used to set spark.network.timeout. See https://spark.apache.org/docs/latest/configuration.html. |
800s | false |
| sleeper.bulk.import.emr.serverless.spark.executor.heartbeat.interval | (The interval between heartbeats from executors to the driver. Used to set spark.executor.heartbeatInterval. See https://spark.apache.org/docs/latest/configuration.html. |
60s | false |
| sleeper.bulk.import.emr.serverless.spark.memory.fraction | The fraction of heap space used for execution and storage. Used to set spark.memory.fraction. See https://spark.apache.org/docs/latest/configuration.html. |
0.80 | false |
| sleeper.bulk.import.emr.serverless.spark.memory.storage.fraction | The amount of storage memory immune to eviction, expressed as a fraction of the heap space used for execution and storage. Used to set spark.memory.storageFraction. See https://spark.apache.org/docs/latest/configuration.html. |
0.30 | false |
| sleeper.bulk.import.emr.serverless.spark.speculation | If true then speculative execution of tasks will be performed. Used to set spark.speculation. See https://spark.apache.org/docs/latest/configuration.html. |
false | false |
| sleeper.bulk.import.emr.serverless.spark.speculation.quantile | Fraction of tasks which must be complete before speculation is enabled for a particular stage. Used to set spark.speculation.quantile. See https://spark.apache.org/docs/latest/configuration.html. |
0.75 | false |
| sleeper.bulk.import.emr.serverless.spark.shuffle.mapStatus.compression.codec | The compression codec for map status results. Used to set spark.shuffle.mapStatus.compression.codec. Stops "Decompression error: Version not supported" errors - only a value of "lz4" has been tested. |
lz4 | false |
| sleeper.bulk.import.emr.serverless.initial.capacity.enabled | Set to enable the pre-initialise capacity option for EMR Serverless application. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
false | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.executor.count | The number of executors to pre-initialise. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
72 | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.executor.cores | The amount of CPUs per executor for the pre-initialise capacity. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
4vCPU | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.executor.memory | The amount of memory per executor for the pre-initialise capacity. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
18GB | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.executor.disk | The amount of storage per executor for the pre-initialise capacity. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
200GB | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.driver.count | The number of drivers to pre-initialise. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
5 | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.driver.cores | The amount of CPUs per driver for the pre-initialise capacity. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
4vCPU | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.driver.memory | The amount of memory per driver for the pre-initialise capacity. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
18GB | true |
| sleeper.bulk.import.emr.serverless.initial.capacity.driver.disk | The amount of storage per driver for the pre-initialise capacity. See: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html |
20GB | true |
| sleeper.default.table.bulk.import.emr.release.label | (Non-persistent EMR mode only) The default EMR release label to be used when creating an EMR cluster for bulk importing data using Spark running on EMR. This property is a default which can be overridden by a table property or by a property in the bulk import job specification. |
emr-7.12.0 | false |
| sleeper.default.table.bulk.import.emr.instance.architecture | (Non-persistent EMR mode only) Which architecture to be used for EC2 instance types in the EMR cluster. Must be either "x86_64" "arm64" or "x86_64,arm64". For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md | arm64 | false |
| sleeper.default.table.bulk.import.emr.master.x86.instance.types | (Non-persistent EMR mode only) The default EC2 x86_64 instance types and weights to be used for the master node of the EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7i.xlarge | false |
| sleeper.default.table.bulk.import.emr.executor.x86.instance.types | (Non-persistent EMR mode only) The default EC2 x86_64 instance types and weights to be used for the executor nodes of the EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7i.4xlarge | false |
| sleeper.default.table.bulk.import.emr.master.arm.instance.types | (Non-persistent EMR mode only) The default EC2 ARM64 instance types and weights to be used for the master node of the EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7g.xlarge | false |
| sleeper.default.table.bulk.import.emr.executor.arm.instance.types | (Non-persistent EMR mode only) The default EC2 ARM64 instance types and weights to be used for the executor nodes of the EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7g.4xlarge | false |
| sleeper.default.table.bulk.import.emr.executor.market.type | (Non-persistent EMR mode only) The default purchasing option to be used for the executor nodes of the EMR cluster. Valid values are ON_DEMAND or SPOT. This property is a default which can be overridden by a table property or by a property in the bulk import job specification. |
SPOT | false |
| sleeper.default.table.bulk.import.emr.executor.initial.capacity | (Non-persistent EMR mode only) The default initial number of capacity units to provision as EC2 instances for executors in the EMR cluster. This is measured in instance fleet capacity units. These are declared alongside the requested instance types, as each type will count for a certain number of units. By default the units are the number of instances. This property is a default which can be overridden by a table property or by a property in the bulk import job specification. |
2 | false |
| sleeper.default.table.bulk.import.emr.executor.max.capacity | (Non-persistent EMR mode only) The default maximum number of capacity units to provision as EC2 instances for executors in the EMR cluster. This is measured in instance fleet capacity units. These are declared alongside the requested instance types, as each type will count for a certain number of units. By default the units are the number of instances. This property is a default which can be overridden by a table property or by a property in the bulk import job specification. |
10 | false |
| sleeper.bulk.import.persistent.emr.release.label | (Persistent EMR mode only) The EMR release used to create the persistent EMR cluster. | emr-7.12.0 | true |
| sleeper.bulk.import.persistent.emr.instance.architecture | (Persistent EMR mode only) Which architecture to be used for EC2 instance types in the EMR cluster. Must be either "x86_64" "arm64" or "x86_64,arm64". For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md | arm64 | false |
| sleeper.bulk.import.persistent.emr.master.x86.instance.types | (Persistent EMR mode only) The EC2 x86_64 instance types and weights used for the master node of the persistent EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7i.xlarge | true |
| sleeper.bulk.import.persistent.emr.executor.x86.instance.types | (Persistent EMR mode only) The EC2 x86_64 instance types and weights used for the executor nodes of the persistent EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7i.4xlarge | true |
| sleeper.bulk.import.persistent.emr.master.arm.instance.types | (Persistent EMR mode only) The EC2 ARM64 instance types and weights used for the master node of the persistent EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7g.xlarge | true |
| sleeper.bulk.import.persistent.emr.executor.arm.instance.types | (Persistent EMR mode only) The EC2 ARM64 instance types and weights used for the executor nodes of the persistent EMR cluster. For more information, see the Bulk import using EMR - Instance types section in docs/usage/bulk-import.md |
m7g.4xlarge | true |
| sleeper.bulk.import.persistent.emr.use.managed.scaling | (Persistent EMR mode only) Whether the persistent EMR cluster should use managed scaling or not. | true | true |
| sleeper.bulk.import.persistent.emr.min.capacity | (Persistent EMR mode only) The minimum number of capacity units to provision as EC2 instances for executors in the persistent EMR cluster. This is measured in instance fleet capacity units. These are declared alongside the requested instance types, as each type will count for a certain number of units. By default the units are the number of instances. If managed scaling is not used then the cluster will be of fixed size, with a number of instances equal to this value. This value must be in the range [1, 2000]. |
1 | true |
| sleeper.bulk.import.persistent.emr.max.capacity | (Persistent EMR mode only) The maximum number of capacity units to provision as EC2 instances for executors in the persistent EMR cluster. This is measured in instance fleet capacity units. These are declared alongside the requested instance types, as each type will count for a certain number of units. By default the units are the number of instances. This value is only used if managed scaling is used. This value must be in the range [1, 2000] and greater than the minimum capacity. |
10 | true |
| sleeper.bulk.import.persistent.emr.step.concurrency.level | (Persistent EMR mode only) This controls the number of EMR steps that can run concurrently. | 2 | true |
| sleeper.bulk.import.persistent.emr.cluster.full.requeue.delay | (Persistent EMR mode only) The number of seconds to wait before requeueing a bulk import job because the persistent EMR cluster is full. | 60 | false |
| sleeper.bulk.import.eks.cluster.admin.roles | (EKS mode only) Names of AWS IAM roles which should have access to administer the EKS cluster. | true | |
| sleeper.bulk.import.eks.api.allowed.security.groups | (EKS mode only) IDs of security groups that should be permitted to reach the EKS Kubernetes API. Each ID listed here is added as an ingress rule on TCP/443 to the cluster's security group. Required if you want to call the API from another host inside the cluster's VPC, since the cluster's default security group only permits traffic from the cluster itself. Leave unset for deployments where only the cluster's own components need API access, or where the API accessed from outside the VPC. |
true | |
| sleeper.bulk.import.eks.is.native.libs.image | (EKS mode only) Set to true if sleeper.bulk.import.eks.repo contains the image built with native Hadoop libraries. By default when deploying with the EKS stack enabled, an image will be built based on the official Spark Docker image, so this should be false. | false | true |
| sleeper.bulk.import.eks.awscli.layer.arn | (EKS mode only) The ARN of a Lambda Layer providing the AWS CLI to use with the EKS kubectl provider. If not set, the default AWS CLI layer included with the CDK will be used. | true | |
| sleeper.bulk.import.eks.cluster.type | (EKS mode only) The type of EKS cluster to deploy for bulk import. Valid values are: [fargate, automode] |
FARGATE | true |
| sleeper.bulk.import.eks.automode.nodepool.instance.types | (EKS mode only, automode cluster type only) Comma-separated list of AWS EC2 instance types that the Karpenter NodePool is allowed to launch for Spark pods. | m7g.xlarge,m7g.2xlarge,m7g.4xlarge,m7g.8xlarge,m7g.12xlarge,m7g.16xlarge,m7gd.xlarge,m7gd.2xlarge,m7gd.4xlarge,m7gd.8xlarge,m7gd.12xlarge,m7gd.16xlarge,m7i.xlarge,m7i.2xlarge,m7i.4xlarge,m7i.8xlarge,m7i.12xlarge,m7i.16xlarge,m6id.xlarge,m6id.2xlarge,m6id.4xlarge,m6id.8xlarge,m6id.12xlarge,m6id.16xlarge | true |
| sleeper.bulk.import.eks.automode.nodepool.cpu.limit | (EKS mode only, automode cluster type only) The maximum total number of CPU cores the Karpenter NodePool is allowed to provision across all nodes. Must be an integer greater than 0. | 164 | true |
| sleeper.bulk.import.eks.job.concurrency.level | (EKS mode only) This controls the number of Kubernetes jobs that can run concurrently in the bulk import namespace. Enforced by a ResourceQuota on count/jobs.batch. | 2 | true |
| sleeper.bulk.import.eks.automode.fluentbit.logging.enabled | (EKS auto mode only) Whether to deploy a FluentBit DaemonSet to collect container logs from EKS auto mode nodes and send them to CloudWatch Logs. Disable this if you manage log collection separately or need to avoid the aws-for-fluent-bit image dependency. | true | true |
| sleeper.bulk.import.eks.spark.executor.instances | (EKS mode only) The number of Spark executors. Used to set spark.executor.instances. See https://spark.apache.org/docs/latest/configuration.html. |
29 | false |
| sleeper.bulk.import.eks.spark.driver.memory | (EKS mode only) The amount of memory allocated to the Spark driver. Used to set spark.driver.memory. Default values are overridden because Fargate doesn't work with Spark's default values. See https://spark.apache.org/docs/latest/configuration.html. |
16g | false |
| sleeper.bulk.import.eks.spark.executor.memory | (EKS mode only) The amount of memory allocated to a Spark executor. Used to set spark.executor.memory. Default values are overridden because Fargate doesn't work with Spark's default values. See https://spark.apache.org/docs/latest/configuration.html. |
16g | false |
| sleeper.bulk.import.eks.spark.driver.memory.overhead | (EKS mode only) The memory overhead for the Spark driver. Used to set spark.driver.memoryOverhead. Fargate provides extra memory so no need to include extra which also messes up the scheduler. See https://spark.apache.org/docs/latest/configuration.html. |
1706m | false |
| sleeper.bulk.import.eks.spark.executor.memory.overhead | (EKS mode only) The memory overhead for a Spark executor. Used to set spark.executor.memoryOverhead. Fargate provides extra memory so no need to include extra which also messes up the scheduler. See https://spark.apache.org/docs/latest/configuration.html. |
1706m | false |
| sleeper.bulk.import.eks.spark.hadoop.fs.s3a.experimental.input.fadvise | (EKS mode only) The S3A input read policy. Used to set spark.hadoop.fs.s3a.experimental.input.fadvise. | sequential | false |
| sleeper.bulk.import.eks.spark.executor.cores | (EKS mode only) The number of cores used by a Spark executor. Used to set spark.executor.cores. Should reflect the Fargate task shape rather than the EMR EC2 instance type. See https://spark.apache.org/docs/latest/configuration.html. |
5 | false |
| sleeper.bulk.import.eks.spark.driver.cores | (EKS mode only) The number of cores used by the Spark driver. Used to set spark.driver.cores. Should reflect the Fargate task shape rather than the EMR EC2 instance type. See https://spark.apache.org/docs/latest/configuration.html. |
5 | false |
| sleeper.bulk.import.eks.spark.default.parallelism | (EKS mode only) The default parallelism for the Spark job. Used to set spark.default.parallelism. Should scale with the total cores across the EKS cluster, which may differ from EMR. See https://spark.apache.org/docs/latest/configuration.html. |
290 | false |
| sleeper.bulk.import.eks.spark.sql.shuffle.partitions | (EKS mode only) The number of partitions used in a Spark SQL/dataframe shuffle operation. Used to set spark.sql.shuffle.partitions. See https://spark.apache.org/docs/latest/configuration.html. |
290 | false |
| sleeper.bulk.import.eks.spark.dynamic.allocation.enabled | (EKS mode only) Whether Spark should use dynamic allocation to scale resources up and down. Used to set spark.dynamicAllocation.enabled. Kubernetes support for dynamic allocation is more limited than YARN's; consider leaving this disabled on EKS. See https://spark.apache.org/docs/latest/configuration.html. |
false | false |
| sleeper.bulk.import.eks.spark.executor.extra.java.options | JVM options passed to the executors. Used to set spark.executor.extraJavaOptions. See https://spark.apache.org/docs/latest/configuration.html. |
-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' | false |
| sleeper.bulk.import.eks.spark.driver.extra.java.options | JVM options passed to the driver. Used to set spark.driver.extraJavaOptions. See https://spark.apache.org/docs/latest/configuration.html. |
-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' | false |
| sleeper.bulk.import.eks.spark.executor.heartbeat.interval | The interval between heartbeats from executors to the driver. Used to set spark.executor.heartbeatInterval. See https://spark.apache.org/docs/latest/configuration.html. |
60s | false |
| sleeper.bulk.import.eks.spark.network.timeout | The default timeout for network interactions in Spark. Used to set spark.network.timeout. See https://spark.apache.org/docs/latest/configuration.html. |
800s | false |
| sleeper.bulk.import.eks.spark.memory.fraction | The fraction of heap space used for execution and storage. Used to set spark.memory.fraction. See https://spark.apache.org/docs/latest/configuration.html. |
0.80 | false |
| sleeper.bulk.import.eks.spark.memory.storage.fraction | The amount of storage memory immune to eviction, expressed as a fraction of the heap space used for execution and storage. Used to set spark.memory.storageFraction. See https://spark.apache.org/docs/latest/configuration.html. |
0.30 | false |
| sleeper.bulk.import.eks.spark.storage.level | The storage to use for temporary caching. Used to set spark.storage.level. See https://aws.amazon.com/blogs/containers/best-practices-for-running-spark-on-amazon-eks/ |
MEMORY_AND_DISK_SER | false |
| sleeper.bulk.import.eks.spark.rdd.compress | Whether to compress serialized RDD partitions. Used to set spark.rdd.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | false |
| sleeper.bulk.import.eks.spark.shuffle.compress | Whether to compress map output files. Used to set spark.shuffle.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | false |
| sleeper.bulk.import.eks.spark.shuffle.spill.compress | Whether to compress data spilled during shuffles. Used to set spark.shuffle.spill.compress. See https://spark.apache.org/docs/latest/configuration.html. |
true | false |
| sleeper.bulk.import.eks.spark.executor.ephemeral.storage | (EKS mode only) The amount of ephemeral storage allocated to each Spark executor pod on Fargate. Set as the ephemeral-storage request and limit on the executor container via a Spark pod template, which Fargate uses to provision ephemeral storage for the pod beyond the default 20 GiB. Accepted values 20Gi - 175Gi. See: https://spark.apache.org/docs/latest/running-on-kubernetes.html https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html#fargate-storage |
175Gi | false |