Defect Detection Application (DDA) - Portal Edition

The Defect Detection Application (DDA) is an edge-deployed computer vision solution for quality assurance in discrete manufacturing environments. Originally developed by the AWS EdgeML service team, DDA is now available as an open-source project under the stewardship of the AWS Manufacturing TFC and Auto/Manufacturing IBU.

Table of Contents

• Overview
• Deployment
  • Step 1: Deploy Portal Infrastructure
  • Step 2: Build and Deploy Frontend
  • Step 3: Post-Deployment Setup
  • Step 4: Build DDA Application
  • Step 5: Set Up Accounts
  • Step 6: Create UseCase
  • Step 7: Setting Up Edge Servers
• Deploy DDA Application to Edge Device
• Deployments
• Devices
• Using the Portal
• ML Workflow
• Using ONNX Models
• Remote Device Access (SSH)
• Optional Datasets
• Inference Results Upload
• Edge Device Management
• Troubleshooting
• Project Structure
• Contributing
• License
• Support

Overview

DDA provides real-time defect detection capabilities for manufacturing quality control using computer vision and machine learning. The system runs at the edge using AWS IoT Greengrass, enabling low-latency inference and reducing dependency on cloud connectivity.

Key Benefits

• Real-time Processing: Sub-second inference times for immediate quality feedback
• Edge Deployment: Operates independently of cloud connectivity
• Scalable Architecture: Supports multiple camera inputs and production lines
• ML Model Flexibility: Compatible with various computer vision models
• Manufacturing Integration: RESTful APIs for integration with existing systems
• Portal Management: Centralized multi-tenant admin portal for managing all deployments

Portal Features

Feature	Description
Data Management	Browse & upload training images to S3
Data Accounts	Centralized management of cross-account data access with dropdown selection
Labeling	Create Ground Truth labeling jobs (classification & segmentation)
Manifest Transformation	Auto-transform Ground Truth manifests to DDA-compatible format
Training	SageMaker training with AWS Marketplace algorithm and manifest validation
Compilation	Compile models for edge (x86-64, ARM64)
Components	Manage Greengrass components
Deployments	Deploy models to edge devices with optional Inference Uploader
Inference Uploader	Automatically upload inference results from edge devices to S3
Devices	Monitor IoT Greengrass devices
Settings	Portal configuration and Data Accounts management (PortalAdmin only)

Deployment

Architecture Overview

DDA supports both single-account and multi-account architectures:

Single-Account (Recommended for Getting Started)

• All components in one AWS account
• Simplest setup and management
• Requires running ./deploy-account-role.sh (select Option 1) to create the DDASageMakerExecutionRole

Multi-Account (Recommended for Production)

• Separates Portal, UseCase, and Data accounts
• Better security and governance
• Requires cross-account IAM roles (created by deploy-account-role.sh)

Architecture Components

Portal Account          UseCase Account         Data Account (Optional)
┌──────────────┐       ┌──────────────┐        ┌──────────────┐
│ CloudFront   │       │ SageMaker    │        │ S3 Buckets   │
│ API Gateway  │──────▶│ Greengrass   │◀──────▶│ (Training    │
│ Cognito      │ STS   │ IoT Core     │        │  Data)       │
│ DynamoDB     │       │ S3 Buckets   │        │              │
└──────────────┘       └──────────────┘        └──────────────┘

Portal Account - Central management hub

• DDA Portal web interface, user authentication and RBAC
• Training and compilation job orchestration

UseCase Account - ML workflow execution

• SageMaker training jobs, model compilation for edge deployment
• Greengrass component management
• Can be the same as Portal Account for single-account setups

Data Account (Optional) - Centralized data storage

• Training datasets and inference results from edge devices

Prerequisites

• AWS CLI configured
• Node.js 18+, Python 3.11+
• AWS CDK: npm install -g aws-cdk
• AWS Marketplace subscription (in UseCase Account — see below)

Important: AWS Marketplace Subscription Required

The DDA training algorithm is an AWS Marketplace product. You must subscribe in the UseCase Account (the account where SageMaker runs) before creating training jobs.

1. Sign in to the UseCase Account AWS Console
2. Go to AWS Marketplace - DDA Algorithm
3. Click Continue to Subscribe → Accept Terms
4. Wait for the subscription to become active (usually a few minutes)

For multi-account setups, the subscription must be in the UseCase Account (where SageMaker runs), not the Portal Account or Data Account.

Step 1: Deploy Portal Infrastructure (Portal Account)

1.1 Configure CloudWatch Logs for API Gateway

# 1. Create IAM role for API Gateway CloudWatch Logs
aws iam create-role \
  --role-name APIGatewayCloudWatchLogsRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "apigateway.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# 2. Attach CloudWatch Logs policy
aws iam attach-role-policy \
  --role-name APIGatewayCloudWatchLogsRole \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

# 3. Get the role ARN and set in API Gateway
ROLE_ARN=$(aws iam get-role --role-name APIGatewayCloudWatchLogsRole --query 'Role.Arn' --output text)
aws apigateway update-account \
  --patch-operations op=replace,path=/cloudwatchRoleArn,value=$ROLE_ARN \
  --region us-east-2

Note: This is a one-time setup per AWS account.

1.2 Deploy Portal Infrastructure with CDK

cd edge-cv-portal

# First time only: bootstrap CDK in your account
cd infrastructure
npm install
npx cdk bootstrap
cd ..

# Deploy all stacks
./deploy-infrastructure.sh

Note: cdk bootstrap must be run from edge-cv-portal/infrastructure/ where cdk.json lives. The deploy-infrastructure.sh script handles npm install and cdk deploy for you after that.

What gets deployed: CloudFront, API Gateway, Cognito, DynamoDB, Lambda functions, S3 buckets, IAM roles.

Save the CDK outputs: PortalApiEndpoint, PortalFrontendUrl, CognitoUserPoolId, CognitoClientId.

Step 2: Build and Deploy Frontend

cd edge-cv-portal
./deploy-frontend.sh

This automatically generates config.json from CDK stack outputs, builds the React app, uploads to S3, and invalidates CloudFront cache.

Step 3: Post-Deployment Setup

3.1 Shared Components

When you deploy the portal infrastructure, the following Greengrass components are automatically provisioned and shared with UseCase accounts:

• LocalServer - Core DDA inference component. The backend container packages the NVIDIA Triton Inference Server, which loads compiled models and serves inference; the GStreamer pipeline calls into Triton via the emltriton plugin.
• InferenceUploader - Optional component for uploading inference results to S3

These are automatically shared when you create a UseCase in the portal. The UseCase creation also provisions the DDAPortalComponentAccessPolicy required by edge devices.

3.2 Create Portal Admin User

# Get User Pool ID from CDK output (or use the value from config.json)
USER_POOL_ID=$(aws cloudformation describe-stacks \
  --stack-name EdgeCVPortalAuthStack \
  --query 'Stacks[0].Outputs[?OutputKey==`AuthConfig`].OutputValue' \
  --output text --region us-east-2 | python3 -c "import sys,json; print(json.load(sys.stdin)['userPoolId'])")



aws cognito-idp admin-set-user-password \
  --user-pool-id $USER_POOL_ID \
  --username admin \
  --password <YOUR_SECURE_PASSWORD> \
  --permanent \
  --region $REGION

aws cognito-idp admin-update-user-attributes \
  --user-pool-id $USER_POOL_ID \
  --username admin \
  --user-attributes Name=custom:role,Value=PortalAdmin \
  --region $REGION

Step 4: Build DDA Application (Build Server)

The DDA application is the core Greengrass component that runs inference on edge devices.

4.1 Build and Publish

Before launching the build server, find your public IP to configure SSH access in the security group:

curl -L https://ifconfig.me

Then launch and connect:

# Launch build server (from edge-cv-portal directory)
cd edge-cv-portal
./launch-arm64-build-server.sh --key-name YOUR_KEY_NAME --region REGION

# Set permissions on your key pair
chmod 400 ~/.ssh/YOUR_KEY_NAME.pem

# Connect to the build server
ssh -i ~/.ssh/YOUR_KEY_NAME.pem ubuntu@<PUBLIC_IP>

# Clone the repo and build
mkdir -p ~/workspace
cd ~/workspace
git clone https://github.com/awslabs/DefectDetectionApplication.git
cd DefectDetectionApplication
./setup-build-server.sh
./gdk-component-build-and-publish.sh

Selecting the build target. For ARM64/Jetson builds, pass the architecture and JetPack version so the correct base image, recipe, and component name are used:

./gdk-component-build-and-publish.sh aarch64 4   # JetPack 4.6 (L4T r32.x) -> aws.edgeml.dda.LocalServer.arm64
./gdk-component-build-and-publish.sh aarch64 5   # JetPack 5   (L4T r35.x) -> aws.edgeml.dda.LocalServer.arm64JP5
./gdk-component-build-and-publish.sh aarch64 6   # JetPack 6   (L4T r36.x) -> aws.edgeml.dda.LocalServer.arm64JP6
./gdk-component-build-and-publish.sh x86_64      # x86_64                  -> aws.edgeml.dda.LocalServer.amd64

Each JetPack target builds against its matching L4T base image (JP5: l4t-jetpack:r35.4.1, JP6: l4t-jetpack:r36.3.0) and is published as a distinct component. Deploy the component that matches the device's JetPack version, and compile models for the matching compilation target (Jetson JetPack 4.x / 5.x / 6.x).

Launch script options:

• --key-name KEY (required) - SSH key pair name
• --subnet-id SUBNET - Subnet ID for VPC selection
• --security-group-id SG - Security group ID
• --instance-type TYPE - EC2 instance type (default: m6g.4xlarge)
• --volume-size SIZE - Root volume size in GB (default: 100)
• --region REGION - AWS region (default: us-east-1)
• --iam-profile PROFILE - IAM instance profile name (default: dda-build-role, auto-created if missing)

What the build script does:

• Creates IAM role (dda-build-role) if needed
• Detects architecture (x86_64 or aarch64)
• Builds Docker images (backend and frontend)
• Publishes to AWS Greengrass component repository

Debugging: VERBOSE=1 ./gdk-component-build-and-publish.sh

Publishing without rebuilding (publish-ecr-only.sh). gdk-component-build-and-publish.sh always does a clean rebuild before publishing. If you have already built the images (e.g. the build succeeded but the publish failed because an auth/login session expired), use publish-ecr-only.sh to publish the already-built images without spending ~15-20 minutes on another build:

./publish-ecr-only.sh        # aarch64 JetPack 5 (aws.edgeml.dda.LocalServer.arm64JP5)

It reuses the locally-built flask-app:latest / react-webapp:latest images and the existing custom-build/ staging directory, and runs only the ECR + S3 publish path used for >2 GB artifacts: bump to the next patch version, push the images to ECR, upload the small scripts/compose zip to S3, rewrite the recipe to reference the docker: + S3 artifacts, register the new component version, and tag it for portal discovery. It fails fast if AWS credentials are invalid or the built images/staging dir are missing (run a build first). It currently targets the arm64 JetPack 5 component; edit COMPONENT_NAME/ARCH near the top for other targets.

Notice: Stop the EC2 build server after building to avoid unnecessary costs.

Manual Setup (Optional)

If you prefer manual IAM role and EC2 setup, create a role with trust policy for ec2.amazonaws.com and attach permissions for Greengrass, IoT, S3, EC2, CloudWatch Logs, CloudWatch Metrics, and ECR. See the inline policy in the collapsed section of the original documentation.

Step 5: Set Up Accounts

5.1 UseCase Account (Required)

cd edge-cv-portal
./deploy-account-role.sh

• Option 1: Single-account setup (creates DDASageMakerExecutionRole)
• Option 2: Multi-account UseCase account (creates DDAPortalAccessRole + DDASageMakerExecutionRole)

Save the generated usecase-account-*-config.txt file — you'll upload it when creating a UseCase.

5.2 Data Account (Optional, for multi-account)

If your training data lives in a separate AWS account:

cd edge-cv-portal
./deploy-account-role.sh
# Select option 3: Data Account

Save the generated data-account-*-config.txt file, then register it in the portal:

1. Go to Settings → Data Accounts → Add Data Account
2. Upload the data-account-*-config.txt file to auto-fill the fields
3. Enter a name and click Register

This only needs to be done once per data account. All future UseCases can select it from a dropdown.

Step 6: Create UseCase

1. Go to UseCases → Create UseCase
2. Choose setup type (Single Account or Multi-Account)
3. For multi-account: upload usecase-account-*-config.txt to auto-fill role details
4. Configure data account (select from registered accounts or enter manually)
5. Click Create

This provisions Greengrass components, creates DDAPortalComponentAccessPolicy, and sets up S3 buckets.

Step 7: Setting Up Edge Servers

Edge device setup is a manual process. After building the DDA application, provision edge servers with AWS IoT Greengrass using the scripts in /station_install/setup_station.sh on the device itself.

Prerequisites:

• DDA application built and published (Step 4)
• Device credentials configured using station_install/edge-device-iam-policy.json (see Edge Device IAM Permissions below)

Testing with EC2 Instances

# Launch edge device (from the station_install directory)
cd station_install
./launch-edge-device.sh \
  -n dda-edge-1 \
  -k YOUR_KEY_NAME \
  -c auto

# Set permissions on your key pair
chmod 400 ~/.ssh/YOUR_KEY_NAME.pem

# Connect and setup
ssh -i ~/.ssh/YOUR_KEY_NAME.pem ubuntu@<PUBLIC_IP>
scp -r -i ~/.ssh/YOUR_KEY_NAME.pem ./station_install ubuntu@<PUBLIC_IP>:/tmp/
cd /tmp/station_install
sudo ./setup_station.sh us-east-1 dda-edge-1

Production Deployment (Jetson Devices)

# Copy station_install to Jetson device
scp -r ./station_install <jetson-user>@<jetson-ip>:/tmp/

# SSH and run setup
ssh <jetson-user>@<jetson-ip>
cd /tmp/station_install
sudo ./setup_station.sh <region> <device-name>

The setup script installs system dependencies (Java, Python 3.11, Docker, GStreamer), installs Greengrass Core, provisions the device as an IoT Thing, configures IAM roles, and sets up directory structure.

Edge Device IAM Permissions

Edge devices need IAM permissions for S3, CloudWatch Logs, cross-account access, IoT Core, and Greengrass. Use the policy in station_install/edge-device-iam-policy.json:

aws iam create-user --user-name dda-edge-device
aws iam put-user-policy \
  --user-name dda-edge-device \
  --policy-name DDAEdgeDevicePolicy \
  --policy-document file://station_install/edge-device-iam-policy.json
aws iam create-access-key --user-name dda-edge-device

Deploy DDA Application to Edge Device

Alert: The DDA application (LocalServer component) must be deployed before any other components or models.

All deployments are managed through the portal:

1. Go to Deployments → Create Deployment
2. Select your edge device or device group
3. Add aws.edgeml.dda.LocalServer.<arch> (arm64 or x86_64)
4. Click Deploy
5. Verify on Devices page → select device → Components tab

Deployments

Deployments follow a two-step process, all managed through the portal:

Step 1: Deploy DDA LocalServer (Infrastructure)

1. Deployments → Create Deployment → Add aws.edgeml.dda.LocalServer.<arch> → Deploy

Step 2: Deploy Trained ML Models

1. Deployments → Create Deployment → Add your compiled model component
2. Optionally enable Inference Uploader for S3 sync
3. Deploy

Deployment Options

• Specific Devices: Deploy to individual devices (recommended for testing)
• Thing Groups: Deploy to all devices in an IoT thing group (recommended for production)
• Auto-Included: aws.greengrass.Nucleus and aws.greengrass.LogManager are automatically included

Monitoring Deployments

Go to Deployments page → select deployment to see status (InProgress, Succeeded, Failed, Cancelled), target, components, and per-device status.

Devices

Device Management (via Portal)

• Devices page shows all registered devices with status, platform, architecture, last seen
• Click a device for: Overview, Components, Deployments, Logs, Diagnostics
• Logs are sent to CloudWatch and viewable in the portal
• Diagnostics show connectivity, Greengrass status, component health, storage, memory

Using the Portal

Simplified Workflow

1. Labeling     → Create labeling job → Complete in Ground Truth UI
2. Transform    → Click "Transform Manifest" (auto-fills URIs)
3. Training     → Select job (shows ✓ Transformed) → Start training
4. Compilation  → Select target architecture (ARM64/x86)
5. Deployment   → Create deployment → Optional: Inference Uploader

Data Management

• Upload and organize training images to S3
• Browse datasets with image counts
• Register pre-labeled datasets for quick testing

Labeling

• Create Ground Truth labeling jobs (classification & segmentation)
• Status auto-syncs from SageMaker
• Supports pre-drawn masks for segmentation guidance

Important: Labeling work is completed in the Ground Truth UI, not the DDA Portal.

Manifest Transformation

Ground Truth creates manifests with job-specific attribute names. The portal auto-transforms them to DDA format during training job creation:

• {job-name} → anomaly-label
• {job-name}-metadata → anomaly-label-metadata
• {job-name}-ref → anomaly-mask-ref (segmentation)

Training

• Select Ground Truth job or pre-labeled dataset
• Auto-detect and transform manifests
• Configure model type (Classification, Segmentation, Robust variants)
• Instance types: ml.g4dn.2xlarge (GPU), ml.p3.2xlarge (GPU), ml.m5.xlarge (CPU)
• Optional auto-compile after training

Compilation

• Compile trained models for edge deployment
• Targets: x86_64 CPU, ARM64, Jetson Xavier
• Compiled models are packaged as Greengrass components

ML Workflow

Path A: Ground Truth Labeling (Recommended)

1. Create Labeling Job → Upload images, configure task type
2. Complete Labeling → Workers label in Ground Truth UI
3. Create Training Job → Transform manifest → Start training
4. Compile Model → Select target architecture
5. Deploy to Device → Via portal Deployments page

Path B: Pre-Labeled Datasets (Quick Start)

1. Register Dataset → Data Management → Pre-Labeled Datasets
2. Create Training Job → Select dataset → Start training
3. Compile Model → Select target architecture
4. Deploy to Device → Via portal Deployments page

Using ONNX Models

DDA supports pluggable inference runtimes so a model can run on the ONNX Runtime engine (or native PyTorch) alongside the default SageMaker Neo / DLR path — selected per model. This lets you bring your own ONNX model (e.g. a YOLO object detector) and migrate off Neo/DLR one model at a time, with no change to existing DLR models. See docs/multi-runtime-inference.md for the design.

Runtime selection

Each model package's manifest.json declares which engine loads it via a runtime field:

runtime	Engine	Artifact
dlr (default when absent)	SageMaker Neo / DLR — full backward compatibility	Neo compiled.* + bundled libdlr.so
onnx	ONNX Runtime	model.onnx
pytorch	Native PyTorch (TorchScript)	model.pt

GPU acceleration for ONNX (CUDA + TensorRT execution providers) is available on JetPack 5 and 6; JetPack 4 runs ONNX on CPU only. GPU requires a LocalServer backend image built with the ONNX GPU runtime (see Build requirement for GPU below).

Object detection (YOLO)

ONNX object-detection models add a task field and a detection block. The raw YOLO output tensor (e.g. [1, 84, 8400]) is decoded on-device (threshold → class-wise NMS → boxes scaled to the source image). Detections are returned as bounding boxes drawn onto the output overlay image plus a detections block in the capture metadata.

{
  "runtime": "onnx",
  "runtime_artifact": "model.onnx",
  "task": "object_detection",
  "detection": {
    "num_classes": 80,
    "score_threshold": 0.25,
    "iou_threshold": 0.45,
    "network_input": 640,
    "class_names": ["person", "bicycle", "..."]
  },
  "model_graph": {
    "model_graph_type": "single_stage_model_graph",
    "stages": [{
      "type": "yolo_object_detection",
      "threshold": 0.25,
      "image_width": 640,
      "image_height": 640,
      "image_range_scale": true,
      "normalize": false
    }]
  }
}

Classification/anomaly ONNX models omit task (they default to the anomaly output contract) and use their normal stage type.

Object detection (RF-DETR / DETR-family)

Object detection is architecture-pluggable via a detection.layout selector, so DETR-family models work alongside YOLO. RF-DETR differs from YOLO in output shape and decode:

	YOLO (v5/v8)	RF-DETR (DETR-family)
ONNX outputs	1 tensor [1, 4+C, N]	2 tensors: boxes [1, Q, 4] + logits [1, Q, C]
Box encoding	xywh (center), network-pixel scale	cxcywh, normalized 0..1
Scoring	max class score per anchor	per-query sigmoid (or softmax)
Filtering	score threshold + NMS	NMS-free top-k over query×class

The on-device decoder (rf_detr_object_detection) identifies the boxes vs logits tensors by shape (tolerating exporter output-order differences), does sigmoid/softmax scoring, a flattened top-k, applies score_threshold, converts cxcywh→xyxy, and scales normalized boxes back to the source image. It returns the same detection result contract as YOLO, so overlay rendering and metadata are unchanged.

{
  "runtime": "onnx",
  "runtime_artifact": "model.onnx",
  "task": "object_detection",
  "detection": {
    "layout": "rf_detr",
    "num_classes": 90,
    "score_threshold": 0.5,
    "top_k": 300,
    "network_input": 560,
    "class_names": ["person", "bicycle", "..."]
  },
  "model_graph": {
    "model_graph_type": "single_stage_model_graph",
    "stages": [{
      "type": "rf_detr_object_detection",
      "threshold": 0.5,
      "image_width": 560,
      "image_height": 560,
      "image_range_scale": true,
      "normalize": false
    }]
  }
}

Additional detection keys for RF-DETR: layout: "rf_detr", top_k (default 300), use_softmax (default false → sigmoid), and background_class (optional index to drop when using softmax). NMS-only keys (iou_threshold) are ignored.

Note: the RF-DETR decoder is validated against synthetic two-tensor outputs. A real RF-DETR ONNX export/device run is still pending; exact output tensor names and network_input vary by exporter (order is handled by shape-based tensor ID).

Import via the portal (Smart Import)

1. Put your model.onnx in the UseCase account S3 — or use the ONNX compile target (Compilation page) to export a trained model to model.onnx.
2. Portal → Smart Import (BYOM):
   • Model Type: Object Detection (or Classification)
   • Runtime / export format: ONNX Runtime
   • Detection architecture (object detection only): YOLO or RF-DETR
   • Input image size (e.g. 640), number of classes, optional class names
   • Convert & Import — this writes a device-correct manifest (runtime: onnx, and for detection the task/detection fields including layout) and packages model.onnx.
3. Compile targets / publish as usual, then deploy the model component to the device.

Build requirement for GPU

The default LocalServer backend image installs the CPU ONNX Runtime. GPU (CUDA/TensorRT) needs the from-source GPU runtime, which is built by default for JetPack 5 and 6 by gdk-component-build-and-publish.sh:

./gdk-component-build-and-publish.sh aarch64 6   # JP6: onnxruntime-gpu (CUDA 12.2 / TRT 8.6)
./gdk-component-build-and-publish.sh aarch64 5   # JP5: onnxruntime-gpu (CUDA 11.4 / TRT 8.5)

build-custom.sh defaults ONNXRUNTIME_GPU=1 for JP5/JP6 (the source build adds ~1–2 h and is memory-heavy). For a fast CPU-only image, set ONNXRUNTIME_GPU=0. JetPack 4 is CPU-only regardless. The engine auto-selects TensorRT → CUDA → CPU providers at load, so the same package runs on either a GPU or CPU image.

Remote Device Access (SSH)

You can open an SSH session to an edge device from the portal (Device page → Remote Access tab) using AWS IoT Secure Tunneling — no inbound ports, public IP, or VPN required.

How it works

1. Enable on the Remote Access tab deploys the AWS-managed aws.greengrass.SecureTunneling component to the device (merged with its existing components). Set the SSH login user (ubuntu on EC2, ggc_user for the Greengrass default, or your device user).
2. Open SSH session creates a short-lived tunnel and returns a source access token.
3. Locally, run the AWS IoT local proxy with that token and SSH to localhost:

   export AWSIOT_TUNNEL_ACCESS_TOKEN=<token>
   localproxy -r <region> -s 5555
   ssh -p 5555 <ssh-user>@localhost

Full steps (incl. local-proxy install and on-device log inspection) are in docs/connect-to-device.md.

Security model (important)

Secure Tunneling reaches the device over an outbound WebSocket to AWS IoT — there is no inbound port on the device. Because nothing listens for inbound connections, a security group / IP allowlist does not apply and would give a false sense of control. Access is instead gated by:

• IAM — only principals allowed to call iotsecuretunneling:OpenTunnel (the portal's Devices Lambda role, exercised by authenticated portal users) can open a tunnel.
• Short-lived tunnel tokens — each session token expires automatically (default 60 minutes).
• The device must have an SSH server and a valid login user; disable the component when not in use.

IP-based allowlisting (a security group) only applies if a device is an EC2 instance you SSH to directly by IP — a different path than Secure Tunneling. A future portal enhancement (see docs/device-web-connect-spec.md) may add a browser terminal.

Optional Datasets (Before Model Training)

Pre-labeled datasets let you train models without creating labeling jobs. Register them in Data Management → Pre-Labeled Datasets.

For detailed instructions on Cookie and Alien sample datasets, see datasets/README.md.

Inference Results Upload (Optional)

The Inference Uploader component enables edge devices to automatically upload inference results to S3.

• Automatically provisioned to UseCase accounts during onboarding
• Enable per deployment: add aws.edgeml.dda.InferenceUploader component
• Configurable upload interval (10s to daily)
• Results organized as: s3://bucket/{usecase-id}/{device-id}/{model-id}/YYYY/MM/DD/

See INFERENCE_UPLOADER_SETUP.md for detailed configuration.

Edge Device Management

Edge device provisioning and management is a manual process. The portal handles deployments, but device setup requires SSH access.

Directory Structure on Device

/aws_dda/
├── greengrass/v2/              # Greengrass installation
│   ├── config/
│   ├── logs/
│   └── packages/
├── dda_triton/
│   └── triton_model_repo/      # NVIDIA Triton model repository (compiled models)
├── image-capture/              # Captured images from camera
├── inference-results/          # Inference results
├── em_agent/                   # Edge Manager Agent
└── check-cloudwatch-logging.sh # Diagnostics script

Common Device Operations

# Check Greengrass status
sudo systemctl status greengrass

# Restart Greengrass
sudo systemctl restart greengrass

# View Greengrass logs
tail -f /aws_dda/greengrass/v2/logs/aws.greengrass.Nucleus.log

# View LocalServer logs
tail -f /aws_dda/greengrass/v2/logs/aws.edgeml.dda.LocalServer.log

# Check deployments
aws greengrassv2 list-effective-deployments \
  --core-device-thing-name your-device-name \
  --region us-east-1

Testing Inference (Without Camera - Folder Mode)

Copy test images to the device and run inference directly:

# Copy images to device
scp -i "device-key.pem" -r ./test-images/* ubuntu@device-ip:/aws_dda/image-capture/<folder-name>

# Test classification
curl -X POST http://localhost:5000/api/v1/inference \
  -F "image=@/aws_dda/image-capture/test-image.jpg" \
  -F "model_name=model-cookie-class"

# Test segmentation
curl -X POST http://localhost:5000/api/v1/inference \
  -F "image=@/aws_dda/image-capture/test-image.jpg" \
  -F "model_name=model-cookie-segmentation"

Batch test all images:

for image in /aws_dda/image-capture/*.jpg; do
  echo "Testing: $(basename $image)"
  curl -s -X POST http://localhost:5000/api/v1/inference \
    -F "image=@$image" \
    -F "model_name=model-cookie-class" | python3 -m json.tool
done

Device Health Monitoring

# Check disk space
df -h /aws_dda/

# Check memory
free -h

# Check network connectivity
aws sts get-caller-identity

# CloudWatch diagnostics
/aws_dda/check-cloudwatch-logging.sh us-east-1

# Search logs for errors
grep -i "error\|failed\|exception" /aws_dda/greengrass/v2/logs/*.log

Troubleshooting

Portal Deployment Issues

CDK Bootstrap Error (SSM parameter /cdk-bootstrap/hnb659fds/version not found):

cdk bootstrap aws://YOUR_ACCOUNT_ID/YOUR_REGION

CloudWatch Logs Error (CloudWatch Logs role ARN must be set): Follow the API Gateway CloudWatch setup in Step 1.1.

Frontend Not Accessible (S3 "NoSuchKey" errors):

./deploy-frontend.sh  # Re-deploy frontend

Edge Device Issues

Device Provisioning Fails (DDAPortalComponentAccessPolicy does not exist): Create a UseCase in the portal first — it provisions this policy.

Greengrass Won't Start:

tail -f /aws_dda/greengrass/v2/logs/aws.greengrass.Nucleus.log
sudo systemctl restart greengrass

S3 Access Denied (S3_HEAD_OBJECT_ACCESS_DENIED): Verify device role has DDAPortalComponentAccessPolicy attached:

aws iam list-attached-role-policies --role-name GreengrassV2TokenExchangeRole

Deployment Fails — Can't Pull Docker Image from ECR (GET_ECR_CREDENTIAL_ERROR / Failed to get auth token for docker login):

EcrException: User: .../GreengrassV2TokenExchangeRole is not authorized to perform:
  ecr:GetAuthorizationToken on resource: * because no identity-based policy allows the action

Docker-based components (e.g. aws.edgeml.dda.LocalServer) pull their image from ECR, so the device token-exchange role needs ECR permissions. These are included in the latest DDAPortalComponentAccessPolicy (and re-applied as the ECRComponentAccess inline policy by setup_station.sh). If you provisioned the device before this was added:

# Refresh the managed policy (re-run in the UseCase account)
./edge-cv-portal/deploy-account-role.sh        # updates DDAPortalComponentAccessPolicy

# OR add the permissions directly to the device role:
aws iam put-role-policy --role-name GreengrassV2TokenExchangeRole \
  --policy-name ECRComponentAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      { "Sid": "AllowEcrAuthToken", "Effect": "Allow",
        "Action": ["ecr:GetAuthorizationToken"], "Resource": "*" },
      { "Sid": "AllowEcrImagePull", "Effect": "Allow",
        "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", "ecr:BatchCheckLayerAvailability"],
        "Resource": "arn:aws:ecr:*:<ACCOUNT_ID>:repository/dda/*" }
    ]
  }'

ecr:GetAuthorizationToken does not support resource scoping and must use "*". No device restart is needed — the role change takes effect on the next deployment retry.

Deployment Fails (COMPONENT_VERSION_REQUIREMENTS_NOT_MET):

• Verify component exists in your account
• Check architecture matches (arm64 vs x86_64)

Deployment Fails — Component Store Full (DISK_SPACE_CRITICAL / SizeLimitException):

SizeLimitException: Component store size limit reached:
  12545441763 bytes existing, 3561542 bytes needed, 10000000000 bytes maximum allowed total

The Greengrass component store has hit its size cap (default 10 GB). This is almost always caused by old LocalServer versions piling up — pre-ECR builds embedded the full multi-GB Docker image tar in the artifact zip, and Greengrass keeps an archived (packages/artifacts/) and unarchived (packages/artifacts-unarchived/) copy of every version. A couple of those fill the store. (The new deployment "bytes needed" is often tiny — e.g. ~3.5 MB for an ECR-based component — which itself tells you the store is full of stale data.)

1. Find which versions are on disk and which one is currently deployed:

sudo du -sh /aws_dda/greengrass/v2/packages/artifacts/aws.edgeml.dda.LocalServer.*/* | sort -h
sudo du -sh /aws_dda/greengrass/v2/packages/artifacts-unarchived/aws.edgeml.dda.LocalServer.*/* | sort -h
# Currently-running version:
grep -i "Resolved component" /aws_dda/greengrass/v2/logs/greengrass.log | tail -3

2. Remove the OLD version directories from both artifacts/ and artifacts-unarchived/. Never delete the version that is currently deployed/running (replace <OLD_VERSION> accordingly):

sudo systemctl stop greengrass
sudo rm -rf /aws_dda/greengrass/v2/packages/artifacts/aws.edgeml.dda.LocalServer.arm64/<OLD_VERSION>
sudo rm -rf /aws_dda/greengrass/v2/packages/artifacts-unarchived/aws.edgeml.dda.LocalServer.arm64/<OLD_VERSION>
sudo systemctl start greengrass

Then re-trigger the deployment.

Alternatively, raise the store cap via the nucleus configuration (then let Greengrass garbage-collect unreferenced versions):

"aws.greengrass.Nucleus": {
  "configurationUpdate": { "merge": "{\"componentStoreMaxSizeBytes\":\"30000000000\"}" }
}

3. Prevent recurrence: publish the LocalServer component through the ECR/S3 path so artifacts stay small (a few MB instead of multiple GB). This happens automatically when the packaged artifact exceeds 2 GB — see gdk-component-build-and-publish.sh. Rebuild/publish for the device's architecture (e.g. ./gdk-component-build-and-publish.sh aarch64 4 for JetPack 4) and deploy that version. ECR-based components also require the device token-exchange role to have ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, and s3:GetObject in the device's region.

Inference Endpoint Not Responding:

ps aux | grep LocalServer
tail -f /aws_dda/greengrass/v2/logs/aws.edgeml.dda.LocalServer.log
netstat -tlnp | grep 5000

Docker Permission Errors:

sudo usermod -aG docker ggc_user
sudo systemctl restart greengrass

Frontend Not Accessible on Device (port 3000):

docker ps | grep frontend
# Use SSH tunnel: ssh -i "key.pem" -L 3000:localhost:3000 ubuntu@device-ip

GStreamer Pipeline Issues:

export GST_DEBUG=3
v4l2-ctl --list-devices
gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink

Model Loading (Triton Server)

The DDA backend packages the NVIDIA Triton Inference Server at /opt/tritonserver. Compiled models are placed in the Triton model repository at /aws_dda/dda_triton/triton_model_repo/. Before deploying a model (or when debugging a pipeline crash), verify the model loads cleanly in Triton.

# Test Triton server directly to verify model loading.
cd /opt/tritonserver/bin
./tritonserver --model-repository /aws_dda/dda_triton/triton_model_repo/

# Expected output should show models in READY status:
# +-------------------------------------------+---------+--------+
# | Model                                     | Version | Status |
# +-------------------------------------------+---------+--------+
# | base_model-bd-dda-classification-arm64    | 1       | READY  |
# | marshal_model-bd-dda-classification-arm64 | 1       | READY  |
# | model-bd-dda-classification-arm64         | 1       | READY  |
# +-------------------------------------------+---------+--------+

If a model shows UNAVAILABLE instead of READY, fix the underlying model error (often a Python syntax error in the model's model.py) before deploying or running the pipeline — a model that fails to load in Triton will crash the GStreamer inference pipeline (commonly with a SIGSEGV).

GStreamer / Triton Inference Pipeline

DDA uses GStreamer pipelines for video processing and ML inference. The custom emltriton (Triton inference) and emlcapture (result capture) plugins live in /usr/lib/panoramagst/. Troubleshoot pipeline issues as follows:

# Install GStreamer tools if not available
sudo apt update
sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good

# Set GStreamer plugin path for DDA custom plugins
export GST_PLUGIN_PATH=/usr/lib/panoramagst/

# Enable GStreamer debug logging
export GST_DEBUG=3  # or GST_DEBUG=4 for more verbose output

# Confirm the DDA custom plugins are present and loadable
gst-inspect-1.0 | grep -E "emltriton|emlcapture"
gst-inspect-1.0 emltriton
gst-inspect-1.0 emlcapture

# Test the full DDA inference pipeline with a sample image
gst-launch-1.0 filesrc blocksize=-1 location="/aws_dda/bd-classification/test-anomaly-1.jpg" ! \
  jpegdec idct-method=2 ! \
  videoconvert ! \
  videoflip method=automatic ! \
  capsfilter caps=video/x-raw,format=RGB ! \
  emltriton model-repo=/aws_dda/dda_triton/triton_model_repo \
    server-path=/opt/tritonserver \
    model=model-bd-dda-classification-arm64 \
    metadata='{"sagemaker_edge_core_capture_data_disk_path": "/aws_dda/inference-results/test", "capture_id": "test-pipeline"}' \
    correlation-id=test-pipeline ! \
  jpegenc idct-method=2 quality=100 ! \
  emlcapture buffer-message-id=file-target_/aws_dda/inference-results/test-jpg \
    interval=0 \
    meta=triton_inference_output_overlay:file-target_/aws_dda/inference-results/test-overlay.jpg

Debugging pipeline crashes (SIGSEGV) — work from the simplest case up:

# 1. Verify the model loads in Triton first (see Model Loading above).
cd /opt/tritonserver/bin
./tritonserver --model-repository /aws_dda/dda_triton/triton_model_repo/

# 2. Test a decode-only pipeline (no inference) to rule out the input image.
export GST_PLUGIN_PATH=/usr/lib/panoramagst/
gst-launch-1.0 filesrc location="/aws_dda/cookies/test-anomaly-3.jpg" ! \
  jpegdec ! videoconvert ! jpegenc ! filesink location="/tmp/test-output.jpg"

# 3. Run the full pipeline INSIDE the backend container (recommended), where the
#    Triton libs and plugin paths are already configured.
docker ps | grep backend
docker exec -it <backend-container-name> bash
# Inside the container:
export GST_PLUGIN_PATH=/usr/lib/panoramagst/
gst-launch-1.0 filesrc blocksize=-1 location="/aws_dda/cookies/test-anomaly-3.jpg" ! \
  emexifextract ! jpegdec idct-method=2 ! videoconvert ! videoflip method=automatic ! \
  capsfilter caps=video/x-raw,format=RGB ! \
  emltriton model-repo=/aws_dda/dda_triton/triton_model_repo \
    server-path=/opt/tritonserver model=model-rajat-segmentation \
    metadata='{"capture_id": "test-pipeline"}' correlation-id=test-pipeline ! \
  jpegenc idct-method=2 quality=100 ! \
  emlcapture buffer-message-id=file-target_/aws_dda/inference-results/test-jpg interval=0

Common GStreamer / Triton issues:

• Segmentation fault: usually a model that failed to load in Triton or a corrupted model file — verify the model is READY in Triton first.
• Plugin not found: ensure GST_PLUGIN_PATH includes /usr/lib/panoramagst/.
• Model loading errors: confirm the Triton server is running and the model-repo/model paths are correct.
• Model syntax errors: fix Python syntax errors in the model's model.py (these surface as UNAVAILABLE in Triton).
• Permission errors: check file permissions on input images and output directories.
• Memory issues: monitor system resources during pipeline execution.

CloudWatch LogManager Issues

The portal auto-includes aws.greengrass.LogManager in deployments to upload device logs to CloudWatch. Logs are organized as:

CloudWatch Log Groups:
  /aws/greengrass/GreengrassSystemComponent/{region}/System     # Greengrass system logs
  /aws/greengrass/UserComponent/{region}/{component-name}       # Per-component logs

Log Streams (per device):
  /YYYY/MM/DD/thing/{thing-name}

"Components detected but no logs yet" in the portal:

• LogManager uploads every 5 minutes — wait for the first upload cycle
• Verify LogManager is installed: ls /aws_dda/greengrass/v2/logs/aws.greengrass.LogManager.log
• If no LogManager log file exists, redeploy from the portal (LogManager is auto-included)

LogManager installed but no CloudWatch log groups:

# Check LogManager config has uploadToCloudWatch enabled
grep -A 80 "aws.greengrass.LogManager" /aws_dda/greengrass/v2/config/effectiveConfig.yaml

# Check LogManager logs for errors
tail -30 /aws_dda/greengrass/v2/logs/aws.greengrass.LogManager.log

# Verify device role has CloudWatch Logs permissions
aws iam list-attached-role-policies --role-name GreengrassV2TokenExchangeRole

LogManager running but portal still shows no logs:

• The portal searches for log streams matching /thing/{device-id} — verify your device's thing name matches
• Check CloudWatch directly: aws logs describe-log-groups --log-group-name-prefix "/aws/greengrass" --region us-east-1
• Redeploy from the portal to ensure LogManager config includes all component names

Training Issues

"Caller is not subscribed to the marketplace offering": Subscribe to the DDA algorithm in the UseCase Account (not Portal Account):

1. Sign in to the UseCase Account console
2. Go to AWS Marketplace - DDA Algorithm
3. Click Continue to Subscribe → Accept Terms
4. Wait a few minutes for activation, then retry

"Manifest validation failed": Transform the manifest first using the portal's Transform Manifest button.

"MaxRuntimeExceeded": Increase Max Runtime (recommend 14400-21600s for production datasets).

Labeling job creation fails (UI template can't be accessed):

cd edge-cv-portal
./deploy-account-role.sh  # Select Option 1 to update trust policy

Build Issues

Parallel builds/publishes clobber each other (wrong image published / staging corruption). Do not run two component builds or publishes at the same time from the same working tree. The build tags a single shared local image flask-app:latest, pushes to the shared dda/flask-app ECR repo (image tag = component version), and writes to shared staging dirs (greengrass-build/, custom-build/, .gdk/). Running two at once races:

• A second build can overwrite flask-app:latest in between the first build finishing and its docker tag flask-app:latest <ecr>:<version> step, so the wrong image gets pushed for that version.
• Concurrent runs overwrite each other's greengrass-build//custom-build/ staging and generated gdk-config.json.
• JetPack targets share the dda/flask-app repo with tag = component version; if two components land on the same version number, the ECR tag is overwritten.

Guidance:

• gdk-component-build-and-publish.sh (official publish): run sequentially — finish JP6, then JP5. It hardcodes flask-app:latest and the shared staging dirs, so parallel runs are unsafe even in separate checkouts.
• Only the raw backend image build can be parallelized, and only when each runs in a separate working tree (e.g. git worktree add) and uses a distinct image name so it doesn't touch flask-app:latest, e.g.:

  # in a second worktree, before building:
  sed -i 's/^    image: flask-app$/    image: flask-app-jp5/' src/docker-compose.yaml
  # copy the gitignored edgemlsdk staging into the worktree first:
  cp -r <main-tree>/src/backend/edgemlsdk src/backend/edgemlsdk

  Even then, the shared dda/flask-app ECR repo means the publish step still must be serialized (or use per-JetPack version numbers that never collide).

Project Structure

defect-detection-application/
├── edge-cv-portal/              # Portal application
│   ├── infrastructure/          # AWS CDK stacks
│   ├── backend/                 # Lambda functions (Python)
│   │   ├── functions/           # Lambda handlers
│   │   └── layers/shared/       # Shared utilities
│   ├── frontend/                # React app (TypeScript)
│   │   ├── src/pages/           # Page components
│   │   ├── src/components/      # Shared components
│   │   └── src/services/        # API services
│   ├── deploy-account-role.sh   # Account setup
│   ├── launch-arm64-build-server.sh  # Build server launcher
│   └── deploy-frontend.sh       # Frontend deployment
├── src/                         # DDA edge application
│   ├── backend/                 # Python Flask backend
│   ├── frontend/                # React web interface
│   └── edgemlsdk/               # ML inference SDK
├── station_install/             # Edge device installation
│   ├── setup_station.sh         # Device provisioning script
│   ├── launch-edge-device.sh    # EC2 edge device launcher
│   └── edge-device-iam-policy.json
├── datasets/                    # Sample datasets
├── build-custom.sh              # Custom build logic
├── gdk-component-build-and-publish.sh
├── publish-ecr-only.sh          # Publish already-built images (no rebuild)
└── setup-build-server.sh

Documentation

Document	Description
edge-cv-portal/ADMIN_GUIDE.md	Portal administration guide
edge-cv-portal/SHARED_COMPONENTS.md	Shared Greengrass components
INFERENCE_UPLOADER_SETUP.md	Inference Uploader setup
datasets/README.md	Sample datasets guide

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

This project adheres to the Amazon Open Source Code of Conduct.

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Support

• GitHub Issues: Report bugs and feature requests via GitHub Issues
• Discussions: Join the community discussions for questions and support