🟧 AWS DataSync Terraform Module

Provisions a complete AWS DataSync transfer pipeline — a keystone task plus its source and destination locations (S3 / EFS / NFS / SMB) and an optional on-prem agent — wired end-to-end, verified, and encrypted in transit from a single module call. Built for the AWS provider v6.x.

🧩 Overview

🔄 One task, fully wired. Creates aws_datasync_task plus everything that is meaningless without it: its source and destination locations and (when on-prem) the agent that reaches them.
🗂️ Four location types, one keyed map each. s3, efs, nfs, and smb locations are map(object(...)) collections rendered via for_each — keys are stable caller strings, merged into one key → ARN map the task binds against.
🔗 Bind by key or by ARN. Each side of the task resolves either from a location key created in this module or from a raw ARN of a location managed elsewhere — supply exactly one form per side.
🔐 Integrity by default. verify_mode = "ONLY_FILES_TRANSFERRED", metadata preservation, and in-transit TLS (DataSync ↔ AWS) are the baked-in defaults.
📊 Logging when you wire it. Supply a CloudWatch Log Group ARN and log_level defaults to TRANSFER; omit it and the task safely defaults to OFF rather than failing.
☁️ Agent-optional. Cloud-only S3↔EFS transfers need no agent; NFS/SMB on-prem sources render an aws_datasync_agent from the optional agents map.
🏷️ Tags everywhere. var.tags flows to the task, every location, and every agent, merges with provider default_tags, and the merged set is surfaced as tags_all.

💡 Why it matters: DataSync moves PII between on-prem storage and AWS. A single, secure-by-default module keeps verification, logging, and least-privilege transfer roles consistent across every migration and recurring sync in the estate.

❤️ Support this project

If these Terraform modules have been helpful to you or your organization, I'd appreciate your support in any of the following ways:

⭐ Star this repository to help others discover this Terraform module.
🤝 Connect with me on LinkedIn: linkedin.com/in/microsoftexpert
☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

Whether it's a star, a professional connection, or a coffee, every gesture helps keep these modules actively maintained and continually improving. Thank you for being part of the community!

🗺️ Where this fits in the family

terraform-aws-datasync is a storage-tier consumer — it references buckets, file systems, IAM roles, subnets, security groups, and a log group by ARN, and emits a task ARN that scheduling and monitoring start or watch.

flowchart LR
 s3["terraform-aws-s3-bucket<br/>bucket arn"]
 efs["terraform-aws-efs<br/>file system arn"]
 iam["terraform-aws-iam-role<br/>bucket-access role arn"]
 vpc["terraform-aws-vpc<br/>subnet arn"]
 sg["terraform-aws-security-group<br/>sg arns"]
 log["terraform-aws-cloudwatch-log-group<br/>log group arn"]
 ds["terraform-aws-datasync"]
 eb["terraform-aws-eventbridge<br/>schedule / start"]

 s3 -->|"s3_bucket_arn"| ds
 efs -->|"efs_file_system_arn"| ds
 iam -->|"bucket_access_role_arn (PassRole)"| ds
 vpc -->|"subnet_arn"| ds
 sg -->|"security_group_arns"| ds
 log -->|"cloudwatch_log_group_arn"| ds
 ds -->|"task arn"| eb

 style ds fill:#FF9900,color:#fff,stroke:#cc7a00,stroke-width:2px

🧬 What this module builds

flowchart TD
 subgraph mod["terraform-aws-datasync"]
 task["aws_datasync_task.this<br/>(keystone)<br/>options + schedule + filters + report"]
 s3loc["aws_datasync_location_s3.this<br/>for_each locations.s3"]
 efsloc["aws_datasync_location_efs.this<br/>for_each locations.efs"]
 nfsloc["aws_datasync_location_nfs.this<br/>for_each locations.nfs"]
 smbloc["aws_datasync_location_smb.this<br/>for_each locations.smb"]
 agent["aws_datasync_agent.this<br/>for_each agents (optional)"]
 end

 s3loc -->|"key → arn"| task
 efsloc -->|"key → arn"| task
 nfsloc -->|"key → arn"| task
 smbloc -->|"key → arn"| task
 agent -.->|"agent_arns"| nfsloc
 agent -.->|"agent_arns"| smbloc

 style task fill:#FF9900,color:#fff,stroke:#cc7a00,stroke-width:2px
 style agent stroke-dasharray: 5 5

Resource	Count	Created when
`aws_datasync_task.this`	1	always (keystone)
`aws_datasync_location_s3.this`	0..N	one per `locations.s3` entry
`aws_datasync_location_efs.this`	0..N	one per `locations.efs` entry
`aws_datasync_location_nfs.this`	0..N	one per `locations.nfs` entry
`aws_datasync_location_smb.this`	0..N	one per `locations.smb` entry
`aws_datasync_agent.this`	0..N	one per `agents` entry (optional)

✅ Provider / Versions

Requirement	Version
Terraform	`>= 1.12.0`
`hashicorp/aws`	`>= 6.0, < 7.0`

The module declares only a required_providers block (providers.tf) and inherits the configured provider. There is no provider {} block and no credential variable — credentials resolve through the standard AWS chain at the root/pipeline level (env vars → SSO/shared credentials → assume_role → instance profile / IRSA → OIDC web identity).

🔑 Required IAM Permissions

Least-privilege actions the Terraform execution identity needs to manage this module.

Action	Required for	Notes
`datasync:CreateTask`, `datasync:UpdateTask`, `datasync:DeleteTask`, `datasync:DescribeTask`	Task lifecycle	Core CRUD for the keystone
`datasync:CreateLocationS3`, `datasync:CreateLocationEfs`, `datasync:CreateLocationNfs`, `datasync:CreateLocationSmb`	Location creation	Only the types you populate
`datasync:DeleteLocation`, `datasync:DescribeLocation*`	Location lifecycle / read-back	`DescribeLocationS3` / `Efs` / `Nfs` / `Smb`
`datasync:CreateAgent`, `datasync:DeleteAgent`, `datasync:DescribeAgent`	Optional agent	Only when `agents` is non-empty
`datasync:TagResource`, `datasync:UntagResource`, `datasync:ListTagsForResource`	Tagging	Applied to task, locations, agents
`iam:PassRole`	S3 location & task report	Required on the `bucket_access_role_arn` — passes the bucket-access role to DataSync; scope to that role's ARN
`iam:CreateServiceLinkedRole` (for `datasync.amazonaws.com`)	First-use SLR	Creates `AWSServiceRoleForDataSync` once per account
`ec2:CreateNetworkInterface`, `ec2:DeleteNetworkInterface`, `ec2:DescribeNetworkInterfaces`, `ec2:DescribeSubnets`, `ec2:DescribeSecurityGroups`	EFS / VPC-endpoint ENI placement	DataSync creates ENIs in the EFS mount-target subnet
`logs:DescribeLogGroups`, `logs:PutResourcePolicy`	Task logging to CloudWatch	Lets DataSync write to the supplied log group

⚠️ iam:PassRole is a real module permission here (unlike pure IAM modules). The Terraform identity passes the S3 bucket_access_role_arn — and any task-report bucket_access_role_arn — to DataSync. Scope it to those specific role ARNs.

🔒 The passed S3 bucket-access role is a separate principal trusted by datasync.amazonaws.com. Constrain its trust policy with aws:SourceAccount / aws:SourceArn to avoid the confused-deputy pattern.

📋 AWS Prerequisites

Service-linked role: AWSServiceRoleForDataSync is auto-created on first use (needs iam:CreateServiceLinkedRole for datasync.amazonaws.com). DataSync uses it for task execution.
S3 location role (mandatory for S3 locations): an IAM role assumable by datasync.amazonaws.com granting s3:ListBucket / s3:GetObject / s3:PutObject (and s3:GetObjectTagging / s3:PutObjectTagging when preserving object tags) on the target bucket. Wire bucket_access_role_arn from terraform-aws-iam-role; the Terraform identity needs iam:PassRole on it.
EFS locations: the EFS mount target must already exist in the subnet you reference. DataSync places ENIs in that subnet (subnet_arn + security_group_arns). TLS (in_transit_encryption = "TLS1_2") additionally requires file_system_access_role_arn.
NFS / SMB locations: require a deployed, reachable DataSync agent (on-prem VMware/Hyper-V/KVM/EC2). Activate via an ip_address (port 80 reachable from the runner) or a pre-fetched activation_key. Cloud-only S3↔EFS transfers need no agent.
CloudWatch logging: a log group plus a resource policy permitting datasync.amazonaws.com to write to it. Wire cloudwatch_log_group_arn from terraform-aws-cloudwatch-log-group.
Enhanced task mode: task_mode = "ENHANCED" is S3 → S3 only and exposes richer per-object metrics; it is incompatible with EFS/NFS/SMB locations.
Region: DataSync is regional — there is no us-east-1 global constraint and no region variable. Source and destination must be reachable from the task's Region.
Quotas (per AWS DataSync quotas):
Soft per-account/per-Region limits on tasks, task executions, and agents (raisable via Service Quotas).
A single location can have up to four Basic-mode and four Enhanced-mode agents assigned.
Per-task throughput, filter-count, and (Basic mode) ~20 million files per execution ceilings apply.

📁 Module Structure

terraform-aws-datasync/
├── providers.tf # required_providers (aws >= 6.0, < 7.0); no provider block
├── variables.tf # name → task → locations → agents → tags → timeouts
├── main.tf # locals (key→ARN merge) + agent / s3 / efs / nfs / smb / task.this
├── outputs.tf # id + arn + name + location/agent maps + tags_all
├── README.md # this file
└── SCOPE.md # in/out-of-scope, IAM permissions, prerequisites, gotchas

⚙️ Quick Start

Smallest working call — a cloud-only S3 → EFS one-shot transfer (no agent), binding locations by key:

module "backup_sync" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-s3-to-efs"

  locations = {
    s3 = {
      src = {
        s3_bucket_arn = module.data_bucket.arn # from terraform-aws-s3-bucket
        subdirectory  = "/exports"
        s3_config     = { bucket_access_role_arn = module.ds_role.arn } # from terraform-aws-iam-role
      }
    }
    efs = {
      dst = {
        efs_file_system_arn = module.archive_efs.arn # from terraform-aws-efs
        subdirectory        = "/incoming"
        ec2_config = {
          subnet_arn          = module.vpc.private_subnet_arns[0] # from terraform-aws-vpc
          security_group_arns = [module.ds_sg.arn]                # from terraform-aws-security-group
        }
      }
    }
  }

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
    cloudwatch_log_group_arn = module.ds_logs.arn # from terraform-aws-cloudwatch-log-group
  }

  tags = {
    Environment = "prod"
    DataClass   = "internal"
  }
}

🔌 Cross-Module Contract

Consumes

Input	Type	Source module
`locations.s3[*].s3_bucket_arn`	`string` (bucket ARN)	`terraform-aws-s3-bucket`
`locations.s3[*].s3_config.bucket_access_role_arn`	`string` (IAM role ARN)	`terraform-aws-iam-role`
`locations.efs[*].efs_file_system_arn`	`string` (EFS ARN)	`terraform-aws-efs`
`locations.efs[*].ec2_config.subnet_arn`	`string` (subnet ARN)	`terraform-aws-vpc`
`locations.efs[*].ec2_config.security_group_arns`	`list(string)` (SG ARNs)	`terraform-aws-security-group`
`locations.smb[*].password`	`string` (sensitive)	`terraform-aws-secrets-manager`
`task.cloudwatch_log_group_arn`	`string` (log group ARN)	`terraform-aws-cloudwatch-log-group`
`task.task_report_config.s3_destination.*`	`string` (bucket + role ARN)	`terraform-aws-s3-bucket` / `terraform-aws-iam-role`

Emits

Output	Description	Consumed by
`id`	Task id (equals the task ARN)	scheduling / EventBridge triggers
`arn`	Task ARN `arn:aws:datasync:<region>:<account>:task/task-xxxx` — the cross-resource reference type	IAM policies, EventBridge rules, monitoring
`name`	Task name (null if AWS generated)	console / CLI references
`source_location_arn`	ARN bound as the task source	wiring / audit
`destination_location_arn`	ARN bound as the task destination	wiring / audit
`location_arns`	Map of `key → location ARN` (all types merged)	task binding, reuse across tasks
`location_ids`	Map of `key → location id` (id equals ARN)	inspection
`s3_location_arns` / `efs_location_arns` / `nfs_location_arns` / `smb_location_arns`	Per-type `key → ARN` maps	targeted wiring
`agent_arns`	Map of `agent key → agent ARN` (empty for cloud-only)	NFS/SMB `agent_arns` wiring
`agent_ids`	Map of `agent key → agent id`	inspection
`tags_all`	All tags incl. provider `default_tags` (resource tags win)	governance / audit

📚 Example Library

1 · Minimal cloud-only S3 → EFS transfer

module "sync" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-s3-efs"

  locations = {
    s3 = {
      src = {
        s3_bucket_arn = "arn:aws:s3:::casey-data"
        subdirectory  = "/exports"
        s3_config     = { bucket_access_role_arn = "arn:aws:iam::123456789012:role/casey-datasync-s3" }
      }
    }
    efs = {
      dst = {
        efs_file_system_arn = "arn:aws:elasticfilesystem:us-east-1:123456789012:file-system/fs-0abc"
        subdirectory        = "/incoming"
        ec2_config = {
          subnet_arn          = "arn:aws:ec2:us-east-1:123456789012:subnet/subnet-0abc"
          security_group_arns = ["arn:aws:ec2:us-east-1:123456789012:security-group/sg-0abc"]
        }
      }
    }
  }

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
  }
}

2 · Bind to a pre-existing location by raw ARN

# Source location is managed by another stack; we only create the destination here.
module "sync" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-into-archive"

  locations = {
    s3 = {
      archive = {
        s3_bucket_arn    = module.archive_bucket.arn
        subdirectory     = "/cold"
        s3_storage_class = "GLACIER"
        s3_config        = { bucket_access_role_arn = module.ds_role.arn }
      }
    }
  }

  task = {
    source_location_arn      = "arn:aws:datasync:us-east-1:123456789012:location/loc-0existingsrc" # raw ARN
    destination_location_key = "archive"                                                           # key in this module
  }
}

3 · Scheduled recurring sync (cron)

module "nightly" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name      = "casey-nightly-archive"
  locations = local.s3_pair # two S3 locations keyed "src" / "dst"

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"

    schedule = {
      schedule_expression = "cron(0 5 ? * MON-FRI *)" # 05:00 UTC weekdays
      status              = "ENABLED"
    }
  }
}

4 · Include / exclude filters

module "filtered" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name      = "casey-filtered"
  locations = local.s3_pair

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"

    includes = { value = "/reports|/statements" } # only these subtrees
    excludes = { value = "*/tmp/*|*.bak" }        # minus these patterns
  }
}

5 · Task logging to CloudWatch (wired from terraform-aws-cloudwatch-log-group)

module "ds_logs" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-cloudwatch-log-group?ref=v1.0.0"
  name   = "/aws/datasync/casey"
}

module "logged_sync" {
  source    = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"
  name      = "casey-logged"
  locations = local.s3_pair

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
    cloudwatch_log_group_arn = module.ds_logs.arn # log_level defaults to TRANSFER when set
  }
}

6 · NFS source with an agent (on-prem → S3)

module "onprem_nfs" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-nfs-to-s3"

  agents = {
    dc1 = {
      ip_address          = "10.20.30.40" # appliance reachable on port 80
      security_group_arns = [module.ds_sg.arn]
      subnet_arns         = [module.vpc.private_subnet_arns[0]]
      vpc_endpoint_id     = module.ds_endpoint.id
    }
  }

  locations = {
    nfs = {
      src = {
        server_hostname = "nas.corp.local"
        subdirectory    = "/export/share"
        on_prem_config  = { agent_arns = [module.onprem_nfs.agent_arns["dc1"]] }
        mount_options   = { version = "NFS4_1" }
      }
    }
    s3 = {
      dst = {
        s3_bucket_arn = module.landing_bucket.arn
        subdirectory  = "/from-nas"
        s3_config     = { bucket_access_role_arn = module.ds_role.arn }
      }
    }
  }

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
  }
}

7 · SMB source with a domain credential (password from Secrets Manager)

variable "smb_password" {
  type      = string
  sensitive = true # source from terraform-aws-secrets-manager, never hard-code
}

module "smb_sync" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name   = "casey-smb-to-efs"
  agents = { dc1 = { activation_key = var.smb_agent_key } }

  locations = {
    smb = {
      src = {
        server_hostname = "fileserver.corp.local"
        subdirectory    = "/share/finance"
        user            = "svc-datasync"
        password        = var.smb_password
        domain          = "CORP"
        agent_arns      = [module.smb_sync.agent_arns["dc1"]]
        mount_options   = { version = "SMB3" }
      }
    }
    efs = {
      dst = {
        efs_file_system_arn = module.efs.arn
        ec2_config = {
          subnet_arn          = module.vpc.private_subnet_arns[0]
          security_group_arns = [module.ds_sg.arn]
        }
      }
    }
  }

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
  }
}

8 · EFS location with TLS in-transit encryption

module "tls_efs" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-tls-efs"

  locations = {
    efs = {
      secure = {
        efs_file_system_arn         = module.efs.arn
        access_point_arn            = module.efs.access_point_arns["app"]
        file_system_access_role_arn = module.efs_access_role.arn # required for TLS
        in_transit_encryption       = "TLS1_2"
        ec2_config = {
          subnet_arn          = module.vpc.private_subnet_arns[0]
          security_group_arns = [module.ds_sg.arn]
        }
      }
    }
    s3 = { dst = { s3_bucket_arn = module.bucket.arn, subdirectory = "/", s3_config = { bucket_access_role_arn = module.ds_role.arn } } }
  }

  task = { source_location_key = "secure", destination_location_key = "dst" }
}

9 · Enhanced task mode (S3 → S3 only)

module "enhanced" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-cross-bucket"

  locations = {
    s3 = {
      src = { s3_bucket_arn = module.bucket_a.arn, subdirectory = "/", s3_config = { bucket_access_role_arn = module.ds_role.arn } }
      dst = { s3_bucket_arn = module.bucket_b.arn, subdirectory = "/", s3_config = { bucket_access_role_arn = module.ds_role.arn } }
    }
  }

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
    task_mode                = "ENHANCED" # richer metrics; S3↔S3 only
  }
}

10 · Task report written to S3

module "reported" {
  source    = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"
  name      = "casey-reported"
  locations = local.s3_pair

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"

    task_report_config = {
      output_type  = "STANDARD"
      report_level = "SUCCESSES_AND_ERRORS"
      s3_destination = {
        bucket_access_role_arn = module.report_role.arn # role writable to the report bucket
        s3_bucket_arn          = module.report_bucket.arn
        subdirectory           = "/datasync-reports"
      }
    }
  }
}

11 · Throttled transfer with NEVER-overwrite safety

module "safe" {
  source    = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"
  name      = "casey-safe-copy"
  locations = local.s3_pair

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"

    options = {
      bytes_per_second       = 104857600 # cap at 100 MB/s to protect the link
      overwrite_mode         = "NEVER"   # never clobber existing destination files
      preserve_deleted_files = "PRESERVE"
      verify_mode            = "POINT_IN_TIME_CONSISTENT"
    }
  }
}

12 · Tags (merge with provider default_tags)

# Caller's provider block owns default_tags; the module never sets it.
provider "aws" {
  default_tags { tags = { Owner = "platform", ManagedBy = "terraform" } }
}

module "tagged" {
  source    = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"
  name      = "casey-tagged"
  locations = local.s3_pair

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
    tags                     = { Tier = "task" } # per-task tags win over var.tags
  }

  tags = {
    Environment = "prod" # resource tag — wins over default_tags on key conflict
    DataClass   = "confidential"
  }
}

# module.tagged.tags_all == { Owner, ManagedBy, Environment, DataClass, Tier }

13 · Customer-managed KMS on the underlying storage

# DataSync inherits encryption from the bucket/file system it reads/writes — there is
# no kms_key_id on the location itself. Encrypt at the storage tier, then point here.
module "cmk" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kms?ref=v1.0.0"
  alias  = "casey/datasync"
}

module "encrypted_bucket" {
  source      = "git::https://github.com/microsoftexpert/terraform-aws-s3-bucket?ref=v1.0.0"
  bucket      = "casey-encrypted-src"
  kms_key_arn = module.cmk.arn # SSE-KMS with the CMK
}

module "sync" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"
  name   = "casey-encrypted-sync"

  locations = {
    s3 = {
      src = {
        s3_bucket_arn = module.encrypted_bucket.arn
        subdirectory  = "/"
        # The bucket-access role MUST also be granted kms:Decrypt / kms:GenerateDataKey on module.cmk.arn
        s3_config = { bucket_access_role_arn = module.ds_role.arn }
      }
      dst = { s3_bucket_arn = module.dst_bucket.arn, subdirectory = "/", s3_config = { bucket_access_role_arn = module.ds_role.arn } }
    }
  }

  task = { source_location_key = "src", destination_location_key = "dst" }
}

14 · Secure-by-default opt-out — verification & logging OFF (exception, label it)

# ⚠️ EXCEPTION: disables integrity verification and logging. Only for throwaway/dev
# copies of non-PII data. Document the justification in your root module.
module "fast_dev_copy" {
  source    = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"
  name      = "dev-fast-copy"
  locations = local.s3_pair

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"

    options = {
      verify_mode = "NONE" # ← opt-out of integrity check (discouraged)
      log_level   = "OFF"  # ← opt-out of logging (discouraged)
    }
  }
}

15 · End-to-end composition — on-prem NFS → encrypted S3, scheduled & logged

# Customer-managed CMK for the landing bucket
module "cmk" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kms?ref=v1.0.0"
  alias  = "casey/migration"
}

# Encrypted landing bucket
module "landing" {
  source      = "git::https://github.com/microsoftexpert/terraform-aws-s3-bucket?ref=v1.0.0"
  bucket      = "casey-migration-landing"
  kms_key_arn = module.cmk.arn
}

# Least-privilege bucket-access role (trusts datasync.amazonaws.com; granted s3:* on the bucket + kms on the CMK)
module "ds_role" {
  source              = "git::https://github.com/microsoftexpert/terraform-aws-iam-role?ref=v1.0.0"
  name                = "casey-datasync-s3"
  assume_role_policy  = data.aws_iam_policy_document.datasync_trust.json
  managed_policy_arns = [module.ds_s3_policy.arn]
}

# Networking for the agent ENIs + log target
module "ds_logs" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-cloudwatch-log-group?ref=v1.0.0"
  name   = "/aws/datasync/casey-migration"
}

# This module — agent + NFS source + S3 destination + schedule + report
module "migration" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-datasync?ref=v1.0.0"

  name = "casey-nas-migration"

  agents = {
    dc1 = {
      ip_address          = "10.20.30.40"
      security_group_arns = [module.ds_sg.arn]                  # from terraform-aws-security-group
      subnet_arns         = [module.vpc.private_subnet_arns[0]] # from terraform-aws-vpc
      vpc_endpoint_id     = module.ds_endpoint.id               # from terraform-aws-vpc-endpoint
    }
  }

  locations = {
    nfs = {
      src = {
        server_hostname = "nas.corp.local"
        subdirectory    = "/export/finance"
        on_prem_config  = { agent_arns = [module.migration.agent_arns["dc1"]] }
        mount_options   = { version = "NFS4_1" }
      }
    }
    s3 = {
      dst = {
        s3_bucket_arn = module.landing.arn
        subdirectory  = "/from-nas"
        s3_config     = { bucket_access_role_arn = module.ds_role.arn }
      }
    }
  }

  task = {
    source_location_key      = "src"
    destination_location_key = "dst"
    cloudwatch_log_group_arn = module.ds_logs.arn

    schedule = { schedule_expression = "cron(0 6 ? * SUN *)" } # weekly catch-up

    options = {
      verify_mode    = "ONLY_FILES_TRANSFERRED" # secure default (explicit here)
      overwrite_mode = "ALWAYS"
    }

    task_report_config = {
      report_level = "SUCCESSES_AND_ERRORS"
      s3_destination = {
        bucket_access_role_arn = module.ds_role.arn
        s3_bucket_arn          = module.landing.arn
        subdirectory           = "/reports"
      }
    }
  }

  tags = { Environment = "prod", DataClass = "confidential", Project = "nas-exit" }
}

📥 Inputs

Name	Type	Default	Description
`name`	`string`	`null`	DataSync task name (keystone). Recommended for identifiability.
`task`	`object({...})`	— required	Task config: source/destination binding (key or ARN per side), `task_mode`, `options`, `schedule`, `includes`/`excludes`, `task_report_config`, per-task `tags`.
`locations`	`object({ s3, efs, nfs, smb })` of `map(object(...))`	`{}`	Locations to create, grouped by type; keys unique across all four maps.
`agents`	`map(object(...))`	`{}`	Optional DataSync agents (NFS/SMB only); exactly one of `activation_key`/`ip_address` each.
`tags`	`map(string)`	`{}`	Tags for all taggable resources (merge with `default_tags`).
`timeouts`	`object({ create })`	`{}`	Optional create timeout (task/agent support `create` only).

See variables.tf for the full heredoc schemas, optional defaults, and the ~25 validation rules (enum checks, the atime/mtime pairing rule, exactly-one binding per side, and cross-map key uniqueness).

🧾 Outputs

Name	Description
`id`	Task id (equals the task ARN).
`arn`	Task ARN (cross-resource reference type).
`name`	Task name (null if AWS generated).
`source_location_arn` / `destination_location_arn`	ARNs bound as the task source / destination.
`location_arns` / `location_ids`	Merged `key → ARN` / `key → id` maps across all location types.
`s3_location_arns` / `efs_location_arns` / `nfs_location_arns` / `smb_location_arns`	Per-type `key → ARN` maps.
`agent_arns` / `agent_ids`	`agent key → ARN` / `id` maps (empty for cloud-only).
`tags_all`	All tags incl. provider `default_tags`.

🧠 Architecture Notes

ARN format (task): arn:aws:datasync:<region>:<account-id>:task/task-0123456789abcdef0. Locations: arn:aws:datasync:<region>:<account-id>:location/loc-0123456789abcdef0. Agents: .../agent/agent-0123456789abcdef0. DataSync is regional — the ARN carries a region segment.
ID format: the task, location, and agent id attributes each equal their ARN. The module's location_ids / agent_ids maps therefore mirror the *_arns maps.
Binding resolution: local.location_arns merges all four location maps into one key → ARN lookup (cross-map key uniqueness is enforced by validation). The task resolves each side via coalesce(raw_arn, lookup(key)); a lifecycle.precondition fails with a clear message if a key doesn't resolve.
Force-new fields: most location identity attributes recreate the location in place — s3_bucket_arn, subdirectory, efs_file_system_arn, NFS/SMB server_hostname, and subdirectory. Recreating a location also re-binds (and may recreate) the task. Edit deliberately.
tags ↔ tags_all ↔ default_tags: var.tags (merged with per-item tags, which win) is applied to the task and every location/agent; tags_all is the provider-computed merge of resource tags over provider default_tags, with resource tags winning on key conflict. default_tags is the caller's provider-block concern — never set inside this module.
Eventual consistency: an agent activated from an ip_address must be running and reachable on port 80 at apply time; activation is one-time. Freshly created IAM/SLR may lag — a task referencing a brand-new bucket_access_role_arn can transiently fail and succeed on retry.
Destroy ordering: EFS locations create ENIs in the mount-target subnet. The DataSync location must be destroyed before its subnet / security group, or destroy fails on dependent ENIs — the resource graph handles this within the module, but a caller deleting the upstream terraform-aws-vpc/terraform-aws-security-group in the same apply must let the location go first.
us-east-1 globals: N/A. DataSync is regional; this module never needs a region-pinned provider alias and declares no region variable.

🧱 Design Principles

Secure-by-default posture and every opt-out, explicitly:

Posture	Default	Opt-out
In-transit encryption (DataSync ↔ AWS)	TLS, always on (service behavior)	n/a
EFS in-transit encryption	caller choice; `TLS1_2` recommended for PII	set `in_transit_encryption` / leave unset
Data verification	`verify_mode = "ONLY_FILES_TRANSFERRED"` (integrity check)	`verify_mode = "NONE"` / `"POINT_IN_TIME_CONSISTENT"`
Metadata preservation	`mtime` / `posix_permissions` / `object_tags` = `PRESERVE`	per-field `options` overrides
Overwrite safety	`overwrite_mode = "ALWAYS"` (mirror)	`overwrite_mode = "NEVER"`
Deleted-file handling	`preserve_deleted_files = "PRESERVE"`	set to `"REMOVE"`
Logging	`log_level = "TRANSFER"` when a log group is supplied	omit the log group (defaults `OFF`) / `log_level = "OFF"`
Least-privilege transfer role	S3 role scoped to the bucket, passed via `iam:PassRole`	caller's role policy

DataSync holds no data at rest of its own — encryption is inherited from the bucket/file system it reads and writes. The secure posture here is integrity verification + in-transit TLS + logging + a least-privilege, bucket-scoped transfer role. Document each opt-out in your root module so reviewers see what was loosened.

Other principles:

One composite, one keystone. The task owns only the resources meaningless without it (its locations and the optional agent). Buckets, file systems, IAM roles, subnets, SGs, and log groups are referenced by ARN, keeping blast radius to DataSync's own resources.
for_each, never count, for every child collection — keyed by stable caller strings so reorders don't churn the plan.
Bind by key or ARN. Locations created here are referenced by key; locations owned elsewhere by raw ARN — exactly one form per side, enforced by validation.
Primary outputs id + arn, plus per-type location/agent maps and tags_all.

🚀 Runbook

# Validate without backend or credentials
terraform init -backend=false
terraform validate
terraform fmt -check

plan / apply require valid AWS credentials (profile / SSO / OIDC) resolved through the standard provider chain, a configured Region, the IAM actions listed above (including iam:PassRole on the S3 role and iam:CreateServiceLinkedRole on first use), and — for NFS/SMB — a reachable, activated agent.

🧪 Testing

terraform init -backend=false && terraform validate — schema + reference integrity (catches binding/precondition wiring).
terraform fmt -check — canonical formatting.
terraform plan against a sandbox account to confirm locations, the optional agent, and the task materialize and bind as expected.
Assert module.<name>.arn, location_arns, agent_arns, and tags_all in your root-module test harness.
For end-to-end validation, start a task execution (StartTaskExecution) and verify the CloudWatch logs and (if configured) the S3 task report.

💬 Example Output

module.sync.aws_datasync_location_s3.this["src"]: Creation complete after 2s [id=arn:aws:datasync:us-east-1:123456789012:location/loc-0aaa]
module.sync.aws_datasync_location_efs.this["dst"]: Creation complete after 3s [id=arn:aws:datasync:us-east-1:123456789012:location/loc-0bbb]
module.sync.aws_datasync_task.this: Creation complete after 4s [id=arn:aws:datasync:us-east-1:123456789012:task/task-0ccc]

Outputs:
arn = "arn:aws:datasync:us-east-1:123456789012:task/task-0ccc"
id = "arn:aws:datasync:us-east-1:123456789012:task/task-0ccc"
location_arns = {
 "src" = "arn:aws:datasync:us-east-1:123456789012:location/loc-0aaa"
 "dst" = "arn:aws:datasync:us-east-1:123456789012:location/loc-0bbb"
}
tags_all = { "DataClass" = "internal", "Environment" = "prod" }

🔍 Troubleshooting

Symptom	Likely cause	Fix
`task.source_location_key did not resolve to a location ARN` (precondition)	Key typo, or location not defined in `var.locations`	Use a key that exists across the four maps, or set `task.source_location_arn`
`Set exactly one of task.source_location_key or task.source_location_arn`	Both (or neither) binding forms supplied for a side	Provide exactly one form per side
`Location keys must be unique across the s3, efs, nfs, and smb maps`	Duplicate key reused in a second location map	Rename so every key is unique across all four maps
`Invalid atime/mtime combination`	`atime`/`mtime` set to an unsupported pair	Use `atime=BEST_EFFORT` + `mtime=PRESERVE`, or `atime=NONE` + `mtime=NONE`
`AccessDenied: iam:PassRole` on apply	TF identity lacks `PassRole` on the S3 `bucket_access_role_arn`	Grant `iam:PassRole` scoped to that role ARN
`Unable to assume role` / S3 access denied at execution	Bucket-access role trust/permissions wrong	Trust `datasync.amazonaws.com`; grant `s3:`/`kms:` as needed on the bucket + CMK
Agent stuck `OFFLINE` / activation fails	Appliance not reachable on port 80, or wrong VPC endpoint	Verify network path/SG; confirm `vpc_endpoint_id` and `subnet_arns`
`InvalidRequestException: enhanced mode requires S3`	`task_mode = "ENHANCED"` with a non-S3 location	Use only S3↔S3, or set `task_mode = "BASIC"`
EFS location create fails	Mount target missing in the referenced subnet	Create the EFS mount target first (upstream `terraform-aws-efs`)
Destroy fails on dependent ENIs	EFS location's ENIs still attached when subnet/SG deleted	Destroy the DataSync location before the subnet/security group
Task fails on `TRANSFER` log level	`cloudwatch_log_group_arn` unset or missing resource policy	Supply the log group + a policy allowing `datasync.amazonaws.com` to write
Tag drift on every plan	A tag also set by provider `default_tags` with a different value	Let resource tags win, or remove the overlap from `default_tags`

🔗 Related Docs

AWS DataSync quotas
Do I need a DataSync agent?
DataSync service-linked roles
Creating a DataSync task report
Terraform: aws_datasync_task · aws_datasync_location_s3 · aws_datasync_location_efs · aws_datasync_location_nfs · aws_datasync_location_smb · aws_datasync_agent
Sibling modules: terraform-aws-s3-bucket, terraform-aws-efs, terraform-aws-iam-role, terraform-aws-vpc, terraform-aws-security-group, terraform-aws-cloudwatch-log-group, terraform-aws-kms, terraform-aws-secrets-manager
Module internals: SCOPE.md

🧡 "Infrastructure as Code should be standardized, consistent, and secure."

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

🟧 AWS DataSync Terraform Module

🧩 Overview

❤️ Support this project

🗺️ Where this fits in the family

🧬 What this module builds

✅ Provider / Versions

🔑 Required IAM Permissions

📋 AWS Prerequisites

📁 Module Structure

⚙️ Quick Start

🔌 Cross-Module Contract

Consumes

Emits

📚 Example Library

📥 Inputs

🧾 Outputs

🧠 Architecture Notes

🧱 Design Principles

🚀 Runbook

🧪 Testing

💬 Example Output

🔍 Troubleshooting

🔗 Related Docs

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
SCOPE.md		SCOPE.md
main.tf		main.tf
outputs.tf		outputs.tf
providers.tf		providers.tf
variables.tf		variables.tf

Folders and files

Latest commit

History

Repository files navigation

🟧 AWS DataSync Terraform Module

🧩 Overview

❤️ Support this project

🗺️ Where this fits in the family

🧬 What this module builds

✅ Provider / Versions

🔑 Required IAM Permissions

📋 AWS Prerequisites

📁 Module Structure

⚙️ Quick Start

🔌 Cross-Module Contract

Consumes

Emits

📚 Example Library

📥 Inputs

🧾 Outputs

🧠 Architecture Notes

🧱 Design Principles

🚀 Runbook

🧪 Testing

💬 Example Output

🔍 Troubleshooting

🔗 Related Docs

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages