Skip to content

Commit 2cc057c

Browse files
author
Anjali Modi
committed
Serverless-pattern for DynamoDB-SQS using EventBridge Pipes
1 parent 8d49555 commit 2cc057c

5 files changed

Lines changed: 243 additions & 0 deletions

File tree

‎eventbridge-pipes-dynamodbstream-to-sqs-serverless/README.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# DynamoDB Stream to SQS queue using EventBridge Pipes
2+
3+
This serverless pattern demonstrates how to send events from DynamoDB Stream to SQS using EventBridge Pipes.
4+
5+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
6+
7+
## Requirements
8+
9+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
10+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
11+
* [Git CLI](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed
12+
* [NodeJS](https://nodejs.org/en/download/) (LTS version) installed
13+
* [Serverless Framework CLI](https://www.serverless.com/framework/docs/getting-started) installed
14+
15+
## Deployment Instructions
16+
17+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
18+
19+
``` sh
20+
git clone https://github.com/aws-samples/serverless-patterns
21+
```
22+
1. Change directory to the pattern directory:
23+
``` sh
24+
cd serverless-patterns/eventbridge-pipes-dynamodbstream-to-sqs-serverless
25+
```
26+
1. From the command line, use npm/yarn to install the development dependencies:
27+
``` sh
28+
npm install
29+
```
30+
-or-
31+
``` sh
32+
yarn install
33+
```
34+
1. From the command line, use Serverless Framework to deploy the AWS resources for the pattern as specified in the serverless.yml file:
35+
``` sh
36+
serverless deploy --verbose
37+
```
38+
The above command will deploy resources to `ap-south-1` region by default. You can override the target region with `--region <region>` CLI option, e.g.
39+
``` sh
40+
serverless deploy --verbose --region us-west-2
41+
```
42+
43+
## How it works
44+
45+
This template will create a DynamoDB table, EventBridge Pipe and a SQS queue. DynamoDB Stream is enabled in the DynamoDB table.
46+
47+
Whenever an item is inserted in DynamoDB table, the stream picks up the event & send it to EventBridge Pipes. The EventBridge Pipes Filter function in the Pipes filters the event and passes to the next states accordingly. For this example pattern, it checks for `INSERT` events & passes it to Enrichment Lambda function if found one. The Lambda function makes some logs regarding to event & event name & finally passes it to the Target SQS queue. In Target SQS queue, when new messages are polled, the event information information can be retrieved.
48+
49+
## Testing
50+
51+
Once this stack is deployed in your AWS account, copy the `DynamoDBSourceTableName` value from the output.
52+
53+
Then, insert one record to the DynamoDB table as follows:
54+
```sh
55+
aws dynamodb put-item \
56+
--table-name DynamoDBSourceTable \
57+
--item \
58+
'{"Album": {"S": "Item-One"}}'
59+
```
60+
When you check the Target SQS queue, you can see the message from DynamoDB stream is available in the SQS queue.
61+
Optional - Under the Enrichment Lambda Function of EventBrudge Pipes, you can see the CloudWatch logs for the event information and eventName.
62+
63+
64+
## Cleanup
65+
1. Delete the stack
66+
```sh
67+
serverless remove --verbose
68+
```
69+
1. Confirm the stack has been deleted
70+
```sh
71+
aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'dynamodbstream-ebpipes-sqs-serverless-prod')].StackStatus"
72+
```
73+
Expected output
74+
```json
75+
[
76+
"DELETE_COMPLETE"
77+
]
78+
```
79+
NOTE: You might need to add `--region <region>` option to AWS CLI command if you AWS CLI default region does not match the one, that you used for the Serverless Framework deployment.
80+
----
81+
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
82+
SPDX-License-Identifier: MIT-0

‎eventbridge-pipes-dynamodbstream-to-sqs-serverless/example-pattern.json‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
{
2+
"title": "DynamoDB Stream to SQS queue using EventBridge Pipes",
3+
"description": "Serverless pattern that sends events from DynamoDB Stream to SQS using EventBridge Pipes. Implemented with serverless framework",
4+
"language": "Node.js",
5+
"level": "200",
6+
"framework": "Serverless Framework",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This serverless pattern demonstrates how to send events from DynamoDB Stream to SQS using EventBridge Pipes. Whenever an item is inserted in DynamoDB table, the stream picks up the event & send it to EventBridge Pipes. ",
11+
"The EventBridge Pipes Filter function in the Pipes filters the event and passes to the next states accordingly. For this example pattern, it checks for INSERT events & passes it to Enrichment Lambda function if found one. The Lambda function makes some logs regarding to event & event name & finally passes it to the Target SQS queue.",
12+
"The intermediate Filter & Enrichment Lambda functions are optional & can be skipped as part of editing the implementation."
13+
]
14+
},
15+
"gitHub": {
16+
"template": {
17+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/sfn-athena-cdk-python",
18+
"templateURL": "serverless-patterns/eventbridge-pipes-dynamodbstream-to-sqs-serverless",
19+
"projectFolder": "eventbridge-pipes-dynamodbstream-to-sqs-serverless",
20+
"templateFile": "eventbridge-pipes-dynamodbstream-to-sqs-serverless/serverless.yml"
21+
}
22+
},
23+
"resources": {
24+
"headline": "Additional resources",
25+
"bullets": [
26+
{
27+
"text": "AWS EventBridge Pipes",
28+
"link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html"
29+
},
30+
{
31+
"text": "serverless-pipes - Serverless Framework plugin",
32+
"link": "https://www.npmjs.com/package/serverless-pipes"
33+
}
34+
]
35+
},
36+
"deploy": {
37+
"text": [
38+
"<code>serverless deploy --verbose</code>"
39+
]
40+
},
41+
"testing": {
42+
"text": [
43+
"See the GitHub repo for detailed testing instructions."
44+
]
45+
},
46+
"cleanup": {
47+
"text": [
48+
"<code>serverless remove --verbose</code>."
49+
]
50+
},
51+
"authors": [
52+
{
53+
"headline": "Presented by Anjali Modi, Senior Software Engineer, Distinction-Dev",
54+
"name": "Anjali Modi",
55+
"image": "https://en.gravatar.com/userimage/147444966/1aad7adb8eca7d94a2f25dbe079bec81.jpeg?size=256",
56+
"bio": "Anjali Modi is a Senior Software Engineer, passionate for developing Serverless solutions at Distinction-Dev, India",
57+
"linkedin":"https://www.linkedin.com/in/anjali-modi-068045119/"
58+
}
59+
]
60+
}

‎eventbridge-pipes-dynamodbstream-to-sqs-serverless/index.js‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
"use strict";
2+
3+
module.exports.handler = async(event) => {
4+
// TODO implement
5+
console.log("enrichment function called")
6+
console.log(JSON.stringify(event[0]));
7+
console.log("Event Name: ", event[0]?.eventName)
8+
return event
9+
};

‎eventbridge-pipes-dynamodbstream-to-sqs-serverless/package.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"name": "eventbridge-pipes-dynamodbstream-to-sqs-serverless",
3+
"version": "1.0.0",
4+
"main": "index.js",
5+
"license": "MIT",
6+
"devDependencies": {
7+
"serverless": "^3.34.0"
8+
},
9+
"dependencies": {
10+
"serverless-pipes": "^1.0.0"
11+
}
12+
}

‎eventbridge-pipes-dynamodbstream-to-sqs-serverless/serverless.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
service: dynamodbstream-ebpipes-sqs-serverless
2+
frameworkVersion: '^3.28.1'
3+
useDotenv: true
4+
5+
custom:
6+
# specify the region using --region flag while deploying the stack, else the default region ap-south-1 will be chosen
7+
region: ${opt:region, "ap-south-1"}
8+
9+
provider:
10+
name: aws
11+
runtime: nodejs16.x
12+
memorySize: 256
13+
timeout: 30
14+
# override the default stage (dev) to be `prod`, or you can use the `--stage` CLI option
15+
stage: ${opt:stage, "prod"}
16+
region: ${self:custom.region}
17+
18+
19+
plugins:
20+
# creates eventbridge-pipes using the below serverless framework plugin.
21+
- serverless-pipes
22+
23+
functions:
24+
pipes-enrichment:
25+
handler: index.handler
26+
27+
28+
# Serverless Framework plugin called as "pipes", used to create EventBridge Pipes
29+
# by providing the required event sources, targets and other parameters as needed.
30+
# Refer `https://github.com/distinction-dev/serverless-pipes` for its usage.
31+
pipes:
32+
testPatternPipes:
33+
enabled: true
34+
source:
35+
dynamodb:
36+
arn:
37+
Fn::GetAtt: [DynamoDBSourceTable, StreamArn]
38+
startingPosition: TRIM_HORIZON
39+
target:
40+
sqs:
41+
arn:
42+
Fn::GetAtt: [TargetSQSQueue, Arn]
43+
enrichment:
44+
name: pipes-enrichment
45+
filter:
46+
- Pattern: "{ \"eventName\": [ \"INSERT\" ]}"
47+
iamRolePipes:
48+
type: "individual"
49+
50+
51+
# creates the source DynamoDB-Stream, target SQS resources & outputs their ARN
52+
resources:
53+
Resources:
54+
DynamoDBSourceTable:
55+
Type: AWS::DynamoDB::Table
56+
Properties:
57+
AttributeDefinitions:
58+
- AttributeName: 'Album'
59+
AttributeType: 'S'
60+
KeySchema:
61+
- AttributeName: 'Album'
62+
KeyType: 'HASH'
63+
ProvisionedThroughput:
64+
ReadCapacityUnits: 5
65+
WriteCapacityUnits: 5
66+
StreamSpecification:
67+
StreamViewType: 'NEW_AND_OLD_IMAGES'
68+
TableName: 'DynamoDBSourceTable'
69+
TargetSQSQueue:
70+
Type: AWS::SQS::Queue
71+
Properties:
72+
QueueName: 'TargetSQSQueue'
73+
Outputs:
74+
DynamoDBSourceTableName:
75+
Value:
76+
Ref: DynamoDBSourceTable
77+
TargetSQSQueueArn:
78+
Value:
79+
Fn::GetAtt: [TargetSQSQueue, Arn]
80+

0 commit comments

Comments
 (0)