Skip to content

Commit 3ee1f2f

Browse files
nareshrajaram2017nareshrajaram2017
committed
lambda-esm-kinesis-filters-terraform:Initial commit before PR
1 parent 41102d0 commit 3ee1f2f

17 files changed

Lines changed: 599 additions & 0 deletions

‎lambda-esm-kinesis-filters-terraform/README.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# AWS Event Source Mapping for Lambda from Amazon Kinesis Data Stream (Terraform)
2+
3+
This pattern demonstrates the ability to filter Amazon Kinesis events so that only a subset of all events is sent to an AWS Lambda function for processing. Demo stack will create a single Amazon Kinesis Data Stream (kinesis_stream_lambda_esm) and two AWS Lambda functions (esm_lambda_with_filter and esm_lambda_with_no_filter), one with specific filter criteria and one without any filtering criterias, that are subscribed to that stream using different filter configurations.
4+
5+
Review [Filter Rule Syntax](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html#filtering-syntax) for more details on the message filtering configuration.
6+
7+
Learn more about this pattern at [Serverless Land Patterns](https://serverlessland.com/patterns/lambda-esm-kinesis-filters-terraform).
8+
9+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
10+
11+
## Requirements
12+
13+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
14+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
15+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
16+
* [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli?in=terraform/aws-get-started) installed
17+
18+
## Deployment Instructions
19+
20+
1. Clone the project to your local working directory
21+
22+
```sh
23+
git clone https://github.com/aws-samples/serverless-patterns/
24+
```
25+
26+
2. Change the working directory to this pattern's directory
27+
28+
```sh
29+
cd serverless-patterns/lambda-esm-kinesis-filters-terraform
30+
```
31+
32+
3. From the command line, initialize terraform to to downloads and installs the providers defined in the configuration:
33+
```
34+
terraform init
35+
```
36+
37+
4. From the command line, apply the configuration in the main.tf file:
38+
```
39+
terraform apply
40+
```
41+
42+
5. During the prompts:
43+
- Enter yes
44+
45+
## How it works
46+
47+
A new Amazon Kinesis Data Stream (kinesis_stream_lambda_esm) is created. Two AWS Lambda functions (esm_lambda_with_filter and esm_lambda_with_no_filter) that are subscribed to that stream with different filter settings. This way we demonstrate how various filtering settings affect which Amazon Kinesis Data Stream events are sent to each AWS Lambda function for processing.
48+
49+
All AWS Lambda functions use the same code for demo purposes.
50+
51+
The following considerations should be taken into account when working with Amazon Kinesis Data Stream events:
52+
53+
- Event payload is base64 encoded and Lambda function is responsible for decoding it before processing
54+
- Event filtering for Amazon Kinesis Data Stream supports a subset of Amazon EventBridge event patterns, for example, Or (multiple fields) and Ends with didn't work
55+
- Filter definition should not contain any whitespace (tabs, space, etc.) in order to work properly!
56+
57+
Note: The default region is `us-east-1`, it can also be changed using the variable `region`.
58+
59+
**Note:** Variables can be supplied in different options, check the [Terraform documentation](https://developer.hashicorp.com/terraform/language/values/variables) for more details.
60+
61+
## Testing
62+
63+
You can execute a test script to submit a number of test messages to the demo Kinesis Data Stream (stream-lambda-esm-filter).
64+
65+
```sh
66+
cd serverless-patterns/lambda-esm-kinesis-filters-terraform/test
67+
python stock.py
68+
```
69+
70+
This script sends a series of test events to kinesis_stream_lambda_esm Kinesis Data Stream. It would generate the following sample events,
71+
72+
{'event_time': '2023-08-31T11:44:06.359748', 'ticker': 'MSFT', 'price': 87.79} <br>
73+
{'event_time': '2023-08-31T11:44:06.502610', 'ticker': 'TBV', 'price': 14.46} <br>
74+
{'event_time': '2023-08-31T11:44:06.527346', 'ticker': 'AMZN', 'price': 65.42} <br>
75+
{'event_time': '2023-08-31T11:44:06.553462', 'ticker': 'INTC', 'price': 48.72} <br>
76+
{'event_time': '2023-08-31T11:44:06.576400', 'ticker': 'AMZN', 'price': 59.14} <br>
77+
78+
## Viewing Test Results
79+
80+
| Log Group | Pattern(s) | Match messages | Comment |
81+
| --- | --- | --- | --- |
82+
| /aws/lambda/esm_lambda_with_no_filter | | ALL | logs all test messages |
83+
| /aws/lambda/esm_lambda_with_filter | {"data" = {"ticker" : ["AMZN"]}} | AMZN | logs events that matches the 'AMZN' ticker symbol only|
84+
85+
## Cleanup
86+
87+
1. Change directory to the pattern directory:
88+
```sh
89+
cd serverless-patterns/lambda-esm-kinesis-filters-terraform
90+
```
91+
92+
2. Delete all created resources
93+
```sh
94+
terraform destroy
95+
```
96+
97+
3. During the prompts:
98+
* Enter yes
99+
100+
4. Confirm all created resources has been deleted
101+
```sh
102+
terraform show
103+
```
104+
105+
## Reference
106+
107+
- [AWS Lambda event source mappings](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html)
108+
- [AWS Lambda event filtering](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html)
109+
110+
----
111+
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
112+
113+
SPDX-License-Identifier: MIT-0

‎lambda-esm-kinesis-filters-terraform/example-pattern.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"title": "AWS Event Source Mapping for Lambda from Amazon Kinesis Data Stream (Terraform)",
3+
"description": "This pattern demonstrates the ability to filter Amazon Kinesis events so that only a subset of all events is sent to an AWS Lambda function for processing.",
4+
"language": "",
5+
"level": "200",
6+
"framework": "Terraform",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"A new Amazon Kinesis Data Stream (kinesis_stream_lambda_esm) is created. Two AWS Lambda functions (esm_lambda_with_filter and esm_lambda_with_no_filter) that are subscribed to that stream with different filter settings.",
11+
"This way we demonstrate how various filtering settings affect which Amazon Kinesis Data Stream events are sent to each AWS Lambda function for processing."
12+
]
13+
},
14+
"gitHub": {
15+
"template": {
16+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-esm-kinesis-filters-terraform",
17+
"templateURL": "serverless-patterns/lambda-esm-kinesis-filters-terraform",
18+
"projectFolder": "lambda-esm-kinesis-filters-terraform",
19+
"templateFile": "lambda-esm-kinesis-filters-terraform/main.tf"
20+
}
21+
},
22+
"resources": {
23+
"bullets": [{
24+
"text": "AWS Event Source Mapping for Lambda from Amazon Kinesis Data Stream (Terraform)",
25+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html"
26+
}
27+
]
28+
},
29+
"deploy": {
30+
"text": [
31+
"terraform init",
32+
"terraform plan",
33+
"terraform apply"
34+
]
35+
},
36+
"testing": {
37+
"text": [
38+
"See the README in the GitHub repo for detailed testing instructions."
39+
]
40+
},
41+
"cleanup": {
42+
"text": [
43+
"terraform destroy",
44+
"terraform show"
45+
]
46+
},
47+
"authors": [{
48+
"name": "Naresh Rajaram",
49+
"image": "",
50+
"bio": "Cloud Infrastructure Architect, AWS",
51+
"linkedin": "https://www.linkedin.com/in/naresh-rajaram-25bb106/"
52+
}]
53+
}

‎lambda-esm-kinesis-filters-terraform/main.tf‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
/* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. */
2+
3+
# --- main.tf ---
4+
5+
# Create Kinesis Stream
6+
module "kinesis_stream" {
7+
source = "./modules/kinesisstream"
8+
}
9+
10+
# Create Lambda Event Filtering
11+
module "esm_filter_lambda" {
12+
source = "./modules/lambda"
13+
stream_arn = module.kinesis_stream.stream_arn
14+
}

‎lambda-esm-kinesis-filters-terraform/modules/kinesisstream/main.tf‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
/* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. */
2+
3+
# --- modules/kinesisstream/main.tf ---
4+
5+
# Create Kinesis Stream
6+
resource "aws_kinesis_stream" "kinesis_stream" {
7+
name = var.name
8+
retention_period = var.retention_period
9+
10+
// PROVISIONED or ON_DEMAND
11+
stream_mode_details {
12+
stream_mode = "ON_DEMAND"
13+
}
14+
}

‎lambda-esm-kinesis-filters-terraform/modules/kinesisstream/outputs.tf‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
/* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. */
2+
3+
# --- modules/kinesisstream/outputs.tf ---
4+
5+
# Kinesis Stream Name
6+
output "stream_name" {
7+
value = aws_kinesis_stream.kinesis_stream.name
8+
description = "Output of Kinesis stream name"
9+
}
10+
11+
# Kinesis Stream ARN
12+
output "stream_arn" {
13+
value = aws_kinesis_stream.kinesis_stream.arn
14+
description = "Output of Kinesis stream ARN"
15+
}
16+

‎lambda-esm-kinesis-filters-terraform/modules/kinesisstream/variables.tf‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
/* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. */
2+
3+
# --- modules/kinesisstream/variables.tf ---
4+
5+
# Name of the Kinesis streams
6+
variable "name" {
7+
description = "name"
8+
type = string
9+
default = "kinesis_stream_lambda_esm"
10+
}
11+
12+
# Kinesis streams - Retention Period
13+
variable "retention_period" {
14+
description = "retention_period"
15+
type = number
16+
default = 24
17+
}
18+

‎lambda-esm-kinesis-filters-terraform/modules/lambda/esm_lambda_with_filter.tf‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
/* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. */
2+
3+
# --- modules/lambda/esm_lambda_with_filter.tf ---
4+
5+
# Zip the source code
6+
data "archive_file" "esm_lambda_with_filter_in_zip_format" {
7+
type = "zip"
8+
source_dir = "${path.module}/src/with_filter"
9+
output_path = "${path.module}/tmp/esm_lambda_with_filter.zip"
10+
}
11+
12+
# Create AWS Lambda - Event Source Mapping (ESM) with filter
13+
resource "aws_lambda_function" "esm_lambda_with_filter" {
14+
filename = "${path.module}/tmp/esm_lambda_with_filter.zip"
15+
function_name = var.esm_lambda_with_filter_function_name
16+
role = aws_iam_role.iam_role_for_esm_lambda_with_filter.arn
17+
handler = var.esm_lambda_with_filter_handler
18+
runtime = var.runtime
19+
timeout = var.timeout
20+
21+
provisioner "local-exec" {
22+
environment = {
23+
file_path = "${path.module}/tmp/esm_lambda_with_filter.zip" //Change this location per your application needs
24+
}
25+
26+
command = "rm -f $file_path"
27+
when = destroy
28+
}
29+
}
30+
31+
32+
# Random number generator
33+
resource "random_id" "random_generator_with_filter" {
34+
byte_length = 8
35+
}
36+
37+
# IAM Role for Lambda
38+
resource "aws_iam_role" "iam_role_for_esm_lambda_with_filter" {
39+
name = "iam_role_for_esm_lambda_with_filter_${random_id.random_generator_with_filter.hex}"
40+
assume_role_policy = file("${path.module}/iam_role_for_lambda.json")
41+
force_detach_policies = true
42+
}
43+
44+
# IAM Policy for Lambda
45+
resource "aws_iam_policy" "iam_policy_for_esm_lambda_with_filter" {
46+
name = "iam_policy_for_esm_lambda_with_filter_${random_id.random_generator_with_filter.hex}"
47+
policy = templatefile("${path.module}/iam_policy_for_lambda.tftpl",
48+
{esm_filter_lambda_arn = aws_lambda_function.esm_lambda_with_filter.arn,
49+
stream_arn = var.stream_arn})
50+
}
51+
52+
# Attach IAM Role with IAM Policy for Lambda
53+
resource "aws_iam_role_policy_attachment" "attach_iam_role_with_iam_policy_with_filter" {
54+
policy_arn = aws_iam_policy.iam_policy_for_esm_lambda_with_filter.arn
55+
role = aws_iam_role.iam_role_for_esm_lambda_with_filter.name
56+
}
57+
58+
# Event Source Mapping
59+
resource "aws_lambda_event_source_mapping" "esm_lambda_with_filter" {
60+
event_source_arn = var.stream_arn
61+
function_name = aws_lambda_function.esm_lambda_with_filter.arn
62+
starting_position = "LATEST"
63+
batch_size = 5 //default - 100
64+
65+
66+
filter_criteria {
67+
filter {
68+
pattern = jsonencode({
69+
data = {
70+
"ticker" : ["AMZN"]
71+
}
72+
})
73+
}
74+
}
75+
}
76+
77+
# Cloudwatch Log Group
78+
resource "aws_cloudwatch_log_group" "esm_lambda_with_filter_log_group" {
79+
name = "/aws/lambda/${aws_lambda_function.esm_lambda_with_filter.function_name}"
80+
retention_in_days = 3
81+
82+
lifecycle {
83+
prevent_destroy = false
84+
}
85+
}

‎lambda-esm-kinesis-filters-terraform/modules/lambda/esm_lambda_with_no_filter.tf‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
/* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. */
2+
3+
# --- modules/lambda/esm_lambda_with_no_filter.tf ---
4+
5+
# Zip the source code
6+
data "archive_file" "esm_lambda_with_no_filter_in_zip_format" {
7+
type = "zip"
8+
source_dir = "${path.module}/src/with_no_filter"
9+
output_path = "${path.module}/tmp/esm_lambda_with_no_filter.zip"
10+
}
11+
12+
# Create AWS Lambda - Event Source Mapping (ESM) with no filter (Basic)
13+
resource "aws_lambda_function" "esm_lambda_with_no_filter" {
14+
filename = "${path.module}/tmp/esm_lambda_with_no_filter.zip"
15+
function_name = var.esm_lambda_with_no_filter_function_name
16+
role = aws_iam_role.iam_role_for_esm_lambda_with_no_filter.arn
17+
handler = var.esm_lambda_with_no_filter_handler
18+
runtime = var.runtime
19+
timeout = var.timeout
20+
21+
provisioner "local-exec" {
22+
environment = {
23+
file_path = "${path.module}/tmp/esm_lambda_with_no_filter.zip" //Change this location per your application needs
24+
}
25+
26+
command = "rm -f $file_path"
27+
when = destroy
28+
}
29+
}
30+
31+
# Random number generator
32+
resource "random_id" "random_generator_no_filter" {
33+
byte_length = 8
34+
}
35+
36+
# IAM Role for Lambda
37+
resource "aws_iam_role" "iam_role_for_esm_lambda_with_no_filter" {
38+
name = "iam_role_for_esm_lambda_with_no_filter_${random_id.random_generator_no_filter.hex}"
39+
assume_role_policy = file("${path.module}/iam_role_for_lambda.json")
40+
force_detach_policies = true
41+
}
42+
43+
# IAM Policy for Lambda
44+
resource "aws_iam_policy" "iam_policy_for_esm_lambda_with_no_filter" {
45+
name = "iam_policy_for_esm_lambda_with_no_filter_${random_id.random_generator_no_filter.hex}"
46+
policy = templatefile("${path.module}/iam_policy_for_lambda.tftpl",
47+
{esm_filter_lambda_arn = aws_lambda_function.esm_lambda_with_filter.arn,
48+
stream_arn = var.stream_arn})
49+
}
50+
51+
# Attach IAM Role with IAM Policy for Lambda
52+
resource "aws_iam_role_policy_attachment" "attach_iam_role_with_iam_policy_no_filter" {
53+
policy_arn = aws_iam_policy.iam_policy_for_esm_lambda_with_no_filter.arn
54+
role = aws_iam_role.iam_role_for_esm_lambda_with_no_filter.name
55+
}
56+
57+
# Event Source Mapping
58+
resource "aws_lambda_event_source_mapping" "esm_with_no_filter" {
59+
event_source_arn = var.stream_arn
60+
function_name = aws_lambda_function.esm_lambda_with_no_filter.arn
61+
starting_position = "LATEST"
62+
batch_size = 5 //default - 100
63+
}
64+
65+
# Cloudwatch log group
66+
resource "aws_cloudwatch_log_group" "esm_with_no_filter_log_group" {
67+
name = "/aws/lambda/${aws_lambda_function.esm_lambda_with_no_filter.function_name}"
68+
retention_in_days = 3
69+
lifecycle {
70+
prevent_destroy = false
71+
}
72+
}

0 commit comments

Comments
 (0)