Skip to content

Commit d687df9

Browse files
julianwoodjulianwood
authored
Merge pull request aws-samples#2918 from marcojahn/main
added lambda-s3-athena-cdk-ts pattern
2 parents b6a1876 + 704b1da commit d687df9

11 files changed

Lines changed: 732 additions & 0 deletions

‎lambda-s3-athena-cdk-ts/.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
!jest.config.js
2+
*.d.ts
3+
node_modules
4+
5+
# CDK asset staging directory
6+
.cdk.staging
7+
cdk.out
8+
9+
# Parcel default cache directory
10+
.parcel-cache
11+
12+
# Mac files
13+
.DS_Store

‎lambda-s3-athena-cdk-ts/README.md‎

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
# AWS Lambda with Amazon S3 Failed-Event Destination and Amazon Athena Analytics
2+
3+
This pattern demonstrates how to use Amazon S3 as a failed-event destination for AWS Lambda asynchronous invocations, with Amazon Athena for analytics on failed events. The pattern includes an AWS Lambda function with business logic that can succeed or fail, automatically capturing failed events to Amazon S3 for analysis.
4+
5+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/lambda-s3-athena-cdk-ts](https://serverlessland.com/patterns/lambda-s3-athena-cdk-ts)
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
## Requirements
10+
11+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
12+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
13+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
14+
* [Node.js and npm](https://nodejs.org/) installed
15+
* [AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) installed
16+
17+
## Deployment Instructions
18+
19+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
20+
21+
```bash
22+
git clone https://github.com/aws-samples/serverless-patterns
23+
```
24+
25+
2. Change directory to the pattern directory:
26+
27+
```bash
28+
cd lambda-s3-athena-cdk-ts
29+
```
30+
31+
3. Install dependencies:
32+
33+
```bash
34+
npm install
35+
```
36+
37+
4. Deploy the CDK stack to your default AWS account and region:
38+
39+
```bash
40+
cdk deploy
41+
```
42+
43+
5. Note the outputs from the CDK deployment process. These contain the resource names and URLs which are used for testing.
44+
45+
## Deployment Outputs
46+
47+
After deployment, CDK will display the following outputs. Save these values for configuration and testing:
48+
49+
| Output Key | Description | Usage |
50+
|------------|-------------|-------|
51+
| `LambdaFunctionName` | AWS Lambda function name | Use this name to invoke the AWS Lambda function via AWS CLI |
52+
| `FailedEventsBucketName` | Amazon S3 bucket storing failed AWS Lambda events | Check this bucket to verify failed events are being captured |
53+
| `AthenaResultsBucketName` | Amazon S3 bucket for Amazon Athena query results | Amazon Athena stores query results here automatically |
54+
| `GlueDatabaseName` | AWS Glue database name (typically `failed_events_db`) | Use in Amazon Athena queries: `FROM <database>.<table>` |
55+
| `GlueTableName` | AWS Glue table name (`failed_events`) | Use in Amazon Athena queries: `FROM <database>.<table>` |
56+
| `AthenaWorkgroupName` | Amazon Athena workgroup name | Specify this workgroup when running Amazon Athena queries |
57+
58+
**Example output:**
59+
```
60+
Outputs:
61+
LambdaS3AthenaCdkStack.LambdaFunctionName = processor-function-abc123
62+
LambdaS3AthenaCdkStack.FailedEventsBucketName = lambdas3athenac-failedeventsbucket12345-abcdef
63+
LambdaS3AthenaCdkStack.AthenaResultsBucketName = lambdas3athenac-athenaresultsbucket67890-ghijkl
64+
LambdaS3AthenaCdkStack.GlueDatabaseName = failed_events_db
65+
LambdaS3AthenaCdkStack.GlueTableName = failed_events
66+
LambdaS3AthenaCdkStack.AthenaWorkgroupName = failed-events-workgroup
67+
```
68+
69+
## How it works
70+
71+
![Architecture Diagram](./lambda-s3-athena-cdk-ts.png)
72+
73+
This pattern creates an AWS Lambda function that implements simple business logic with three actions: `process`, `validate`, and `calculate`. When the AWS Lambda function fails (throws an error) during asynchronous invocation, the failed event is automatically sent to an Amazon S3 bucket configured as the failed-event destination.
74+
75+
The pattern also sets up AWS Glue and Amazon Athena to enable SQL-based analytics on the failed events stored in Amazon S3. This allows you to query and analyze error patterns, identify common failure scenarios, and gain insights into your application's error behavior.
76+
77+
Architecture flow:
78+
1. Client invokes AWS Lambda function asynchronously via AWS CLI
79+
2. AWS Lambda processes the request based on the action type
80+
3. On success: AWS Lambda returns successful response
81+
4. On failure: AWS Lambda throws error, and AWS automatically sends the failed event to Amazon S3
82+
5. Failed events are stored in Amazon S3 as JSON files
83+
6. Amazon Athena queries the failed events using AWS Glue Data Catalog for analytics
84+
85+
### AWS Lambda Destinations and Invocation Types
86+
87+
**Important**: AWS Lambda destinations only work with **asynchronous invocations** and **stream-based invocations** (on-failure only). They do not trigger for synchronous invocations.
88+
89+
| Invocation Type | Examples | Destinations Supported? |
90+
|----------------|----------|------------------------|
91+
| **Synchronous** | Amazon API Gateway, Application Load Balancer, SDK RequestResponse | ❌ No |
92+
| **Asynchronous** | Amazon S3, Amazon SNS, Amazon EventBridge, SDK Event | ✅ Yes |
93+
| **Stream/Poll-based** | Amazon Kinesis, Amazon DynamoDB Streams, Amazon SQS | ✅ On-failure only |
94+
95+
This pattern uses **asynchronous invocation via AWS CLI** (`--invocation-type Event`) to demonstrate the Amazon S3 failed-event destination feature.
96+
97+
The AWS Glue Data Catalog is required for Amazon Athena to query Amazon S3 data, providing the schema definition, data location, and SerDe configuration needed to parse the JSON files. This pattern uses a simplified configuration without a predefined schema, allowing Amazon Athena to perform full scans of all failed event files for learning and low-volume workloads.
98+
99+
## Testing
100+
101+
### Test Successful Requests
102+
103+
1. Get the AWS Lambda function name from the stack outputs:
104+
105+
```bash
106+
FUNCTION_NAME=$(aws cloudformation describe-stacks --stack-name LambdaS3AthenaCdkStack --query 'Stacks[0].Outputs[?OutputKey==`LambdaFunctionName`].OutputValue' --output text)
107+
```
108+
109+
2. Test the `process` action (success) with asynchronous invocation:
110+
111+
```bash
112+
aws lambda invoke \
113+
--function-name $FUNCTION_NAME \
114+
--invocation-type Event \
115+
--cli-binary-format raw-in-base64-out \
116+
--payload '{"action": "process", "value": 10}' \
117+
response.json
118+
```
119+
120+
Expected response: `{"StatusCode": 202}` (request accepted)
121+
122+
3. Test the `validate` action (success):
123+
124+
```bash
125+
aws lambda invoke \
126+
--function-name $FUNCTION_NAME \
127+
--invocation-type Event \
128+
--cli-binary-format raw-in-base64-out \
129+
--payload '{"action": "validate", "value": "hello"}' \
130+
response.json
131+
```
132+
133+
4. Test the `calculate` action (success):
134+
135+
```bash
136+
aws lambda invoke \
137+
--function-name $FUNCTION_NAME \
138+
--invocation-type Event \
139+
--cli-binary-format raw-in-base64-out \
140+
--payload '{"action": "calculate", "value": [1, 2, 3, 4, 5]}' \
141+
response.json
142+
```
143+
144+
### Test Failed Requests
145+
146+
1. Test with invalid value for `process` (negative number):
147+
148+
```bash
149+
aws lambda invoke \
150+
--function-name $FUNCTION_NAME \
151+
--invocation-type Event \
152+
--cli-binary-format raw-in-base64-out \
153+
--payload '{"action": "process", "value": -5}' \
154+
response.json
155+
```
156+
157+
2. Test with invalid value for `validate` (empty string):
158+
159+
```bash
160+
aws lambda invoke \
161+
--function-name $FUNCTION_NAME \
162+
--invocation-type Event \
163+
--cli-binary-format raw-in-base64-out \
164+
--payload '{"action": "validate", "value": ""}' \
165+
response.json
166+
```
167+
168+
3. Test with invalid value for `calculate` (empty array):
169+
170+
```bash
171+
aws lambda invoke \
172+
--function-name $FUNCTION_NAME \
173+
--invocation-type Event \
174+
--cli-binary-format raw-in-base64-out \
175+
--payload '{"action": "calculate", "value": []}' \
176+
response.json
177+
```
178+
179+
4. Test with unknown action:
180+
181+
```bash
182+
aws lambda invoke \
183+
--function-name $FUNCTION_NAME \
184+
--invocation-type Event \
185+
--cli-binary-format raw-in-base64-out \
186+
--payload '{"action": "unknown", "value": "test"}' \
187+
response.json
188+
```
189+
190+
5. Wait a few minutes for the failed events to be written to Amazon S3, then verify they exist:
191+
192+
```bash
193+
BUCKET_NAME=$(aws cloudformation describe-stacks --stack-name LambdaS3AthenaCdkStack --query 'Stacks[0].Outputs[?OutputKey==`FailedEventsBucketName`].OutputValue' --output text)
194+
aws s3 ls s3://$BUCKET_NAME/ --recursive
195+
```
196+
197+
Note: Failed events typically appear in Amazon S3 within 1-2 minutes after the AWS Lambda invocation fails.
198+
199+
### Query Failed Events with Amazon Athena
200+
201+
1. Open the Amazon Athena console to run queries.
202+
* You will need to setup a query result location in Amazon S3 if you've never run a query before
203+
* This can be changed later in the settings
204+
205+
2. Query to count failed events by error type:
206+
207+
```sql
208+
SELECT
209+
responsepayload.errortype as error_type,
210+
COUNT(*) as error_count
211+
FROM failed_events
212+
GROUP BY responsepayload.errortype
213+
ORDER BY error_count DESC;
214+
```
215+
216+
3. Query to see detailed error messages:
217+
218+
```sql
219+
SELECT
220+
timestamp,
221+
responsepayload.errortype as error_type,
222+
responsepayload.errormessage as error_message,
223+
requestpayload.body as request_body
224+
FROM failed_events
225+
ORDER BY timestamp DESC
226+
LIMIT 10;
227+
```
228+
229+
4. Query to analyze errors by action type:
230+
231+
```sql
232+
SELECT
233+
json_extract_scalar(requestpayload.body, '$.action') as action,
234+
responsepayload.errortype as error_type,
235+
COUNT(*) as count
236+
FROM failed_events
237+
GROUP BY
238+
json_extract_scalar(requestpayload.body, '$.action'),
239+
responsepayload.errortype
240+
ORDER BY count DESC;
241+
```
242+
243+
5. Query to find errors within a specific time range:
244+
245+
```sql
246+
SELECT
247+
timestamp,
248+
responsepayload.errormessage as error_message,
249+
requestpayload.body as request_body
250+
FROM failed_events
251+
WHERE timestamp >= '2025-01-01T00:00:00Z'
252+
ORDER BY timestamp DESC;
253+
```
254+
255+
## Cleanup
256+
257+
1. Delete the stack:
258+
259+
```bash
260+
cdk destroy
261+
```
262+
263+
2. Confirm the deletion when prompted.
264+
265+
---
266+
267+
Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
268+
269+
SPDX-License-Identifier: MIT-0

‎lambda-s3-athena-cdk-ts/bin/lambda-s3-athena-cdk-ts.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from 'aws-cdk-lib';
4+
import { PatternStack } from '../lib/pattern-stack';
5+
6+
const app = new cdk.App();
7+
new PatternStack(app, 'LambdaS3AthenaCdkStack', {
8+
env: {
9+
account: process.env.CDK_DEFAULT_ACCOUNT,
10+
region: process.env.CDK_DEFAULT_REGION,
11+
}
12+
});

‎lambda-s3-athena-cdk-ts/cdk.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/lambda-s3-athena-cdk-ts.ts",
3+
"watch": {
4+
"include": ["**"],
5+
"exclude": [
6+
"README.md",
7+
"cdk*.json",
8+
"**/*.d.ts",
9+
"**/*.js",
10+
"tsconfig.json",
11+
"package*.json",
12+
"yarn.lock",
13+
"node_modules",
14+
"cdk.out"
15+
]
16+
},
17+
"context": {
18+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
19+
"@aws-cdk/core:checkSecretUsage": true,
20+
"@aws-cdk/core:target-partitions": ["aws", "aws-cn"]
21+
}
22+
}

‎lambda-s3-athena-cdk-ts/example-pattern.json‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
{
2+
"title": "AWS Lambda Failed-Event Destination and Amazon Athena Analytics",
3+
"description": "Capture AWS Lambda failed events to Amazon S3 and analyze them with Amazon Athena for error insights and patterns using asynchronous invocations.",
4+
"language": "TypeScript",
5+
"level": "200",
6+
"framework": "AWS CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern demonstrates how to use Amazon S3 as a failed-event destination for AWS Lambda asynchronous invocations.",
11+
"A Lambda function implements business logic with success and failure scenarios.",
12+
"When the Lambda function fails during asynchronous invocation, AWS automatically captures the failed event to S3 for analysis.",
13+
"Amazon Athena with AWS Glue enables SQL-based analytics on failed events to identify error patterns and gain insights.",
14+
"The pattern uses AWS CLI with --invocation-type Event to demonstrate asynchronous invocation.",
15+
"Important: Lambda destinations only work with asynchronous invocations (S3, SNS, EventBridge) and stream-based sources (Kinesis, DynamoDB Streams, SQS). They do not trigger for synchronous invocations."
16+
]
17+
},
18+
"gitHub": {
19+
"template": {
20+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-s3-athena-cdk-ts",
21+
"templateURL": "serverless-patterns/lambda-s3-athena-cdk-ts",
22+
"projectFolder": "lambda-s3-athena-cdk-ts",
23+
"templateFile": "lib/pattern-stack.ts"
24+
}
25+
},
26+
"resources": {
27+
"bullets": [
28+
{
29+
"text": "AWS Lambda S3 Failed-Event Destination Announcement",
30+
"link": "https://aws.amazon.com/about-aws/whats-new/2024/11/aws-lambda-s3-failed-event-destination-stream-event-sources/"
31+
},
32+
{
33+
"text": "AWS Lambda Destinations Documentation",
34+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#invocation-async-destinations"
35+
},
36+
{
37+
"text": "Amazon Athena Documentation",
38+
"link": "https://docs.aws.amazon.com/athena/latest/ug/what-is.html"
39+
},
40+
{
41+
"text": "AWS Glue Data Catalog",
42+
"link": "https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html"
43+
}
44+
]
45+
},
46+
"deploy": {
47+
"text": [
48+
"Clone the repository: <code>git clone https://github.com/aws-samples/serverless-patterns</code>",
49+
"Change directory: <code>cd lambda-s3-athena-cdk-ts</code>",
50+
"Install dependencies: <code>npm install</code>",
51+
"Deploy the CDK stack: <code>cdk deploy</code>"
52+
]
53+
},
54+
"testing": {
55+
"text": [
56+
"Get Lambda function name: <code>FUNCTION_NAME=$(aws cloudformation describe-stacks --stack-name LambdaS3AthenaCdkStack --query 'Stacks[0].Outputs[?OutputKey==`LambdaFunctionName`].OutputValue' --output text)</code>",
57+
"Test successful request: <code>aws lambda invoke --function-name $FUNCTION_NAME --invocation-type Event --payload '{\"action\": \"process\", \"value\": 10}' response.json</code>",
58+
"Test failed request: <code>aws lambda invoke --function-name $FUNCTION_NAME --invocation-type Event --payload '{\"action\": \"process\", \"value\": -5}' response.json</code>",
59+
"Verify failed events in S3: <code>aws s3 ls s3://$(aws cloudformation describe-stacks --stack-name LambdaS3AthenaCdkStack --query 'Stacks[0].Outputs[?OutputKey==`FailedEventsBucketName`].OutputValue' --output text)/ --recursive</code>",
60+
"Query errors with Athena: <code>SELECT responsepayload.errortype, COUNT(*) FROM failed_events GROUP BY responsepayload.errortype;</code>"
61+
]
62+
},
63+
"cleanup": {
64+
"text": ["Delete the stack: <code>cdk destroy</code>"]
65+
},
66+
"authors": [
67+
{
68+
"name": "Marco Jahn",
69+
"image": "https://sessionize.com/image/e99b-400o400o2-pqR4BacUSzHrq4fgZ4wwEQ.png",
70+
"bio": "Senior Solutions Architect, Amazon Web Services",
71+
"linkedin": "marcojahn"
72+
}
73+
]
74+
}

0 commit comments

Comments
 (0)