Skip to content

Commit 095531b

Browse files
julianwoodjulianwood
committed
Amend Kafka patterns
1 parent 40ad58d commit 095531b

3 files changed

Lines changed: 124 additions & 112 deletions

File tree

‎msk-lambda-iam-node-sam/README.md‎

Lines changed: 42 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,55 @@
11
# Node.js Lambda Kafka Consumer with IAM Auth, using SAM
22

3-
This pattern is an example of a Lambda function that will consume messages from an Managed Streaming for Kafka(MSK) topic, where the MSK Cluster has been configured to use IAM Authentication. This pattern assumes you already have an MSK Cluster with a Topic configured, if you need a sample pattern to deploy an MSK Cluster either in Provisioned or Serverless modes please see the [msk-cfn-sasl-lambda pattern](https://serverlessland.com/patterns/msk-cfn-sasl-lambda)
3+
This pattern is an example of a Lambda function that consumes messages from an Amazon Managed Streaming for Kafka (Amazon MSK) topic, where the MSK Cluster has been configured to use IAM authentication. This pattern assumes you already have an MSK cluster with a topic configured, if you need a sample pattern to deploy an MSK cluster either in Provisioned or Serverless modes please see the [msk-cfn-sasl-lambda pattern](https://serverlessland.com/patterns/msk-cfn-sasl-lambda).
44

5-
This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. It includes the following files and folders.
5+
This project contains source code and supporting files for a serverless application that you can deploy with the AWS Serverless Application Model (AWS SAM) CLI. It includes the following files and folders.
66

77
- HandlerKafka - Code for the application's Lambda function.
88
- events - Invocation events that you can use to invoke the function.
9-
- template.yaml - A template that defines the application's AWS resources.
9+
- template.yaml - An AWS SAM template that defines the application's AWS resources.
1010

11-
The application creates a Lambda function that will listen to Kafka messages on a topic linked to an MSK Cluster. These resources are defined in the `template.yaml` file in this project. You can update the template to add AWS resources through the same deployment process that updates your application code.
11+
The application creates a Lambda function that listens to Kafka messages on a topic of an MSK Cluster. These resources are defined in the `template.yaml` file in this project. You can update the template to add AWS resources through the same deployment process that updates your application code.
1212

1313
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
1414

1515
## Requirements
1616

1717
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
1818
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
19-
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
19+
* [Git installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
2020
* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed
2121
* Create MSK cluster and topic that will be used for testing. It is important to create the topic before deploying the Lambda function, otherwise the event source mapping will stay disabled.
2222

2323
## Deploy the sample application
2424

25-
The Serverless Application Model Command Line Interface (SAM CLI) is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. It uses Docker to run your functions in an Amazon Linux environment that matches Lambda. It can also emulate your application's build environment and API.
25+
The AWS SAM CLI is a serverless tool for building and testing Lambda applications. It uses Docker to locally test your functions in an Amazon Linux environment that resembles the Lambda execution environment. It can also emulate your application's build environment and API.
2626

27-
To use the SAM CLI, you need the following tools.
27+
To use the AWS SAM CLI, you need the following tools.
2828

29-
* SAM CLI - [Install the SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
30-
* Node.js - [Install Node.js 18](https://nodejs.org/en/), including the NPM package management tool.
29+
* AWS SAM CLI - [Install the AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
3130
* Docker - [Install Docker community edition](https://hub.docker.com/search/?type=edition&offering=community)
3231

33-
To build and deploy your application for the first time, run the following in your shell:
34-
35-
```bash
36-
sam build
37-
sam deploy --guided
38-
```
32+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
33+
```
34+
git clone https://github.com/aws-samples/serverless-patterns.git
35+
```
36+
1. Change directory to the pattern directory:
37+
```
38+
cd msk-lambda-iam-node-sam
39+
```
3940
40-
The first command will build the source of your application. The second command will package and deploy your application to AWS, with a series of prompts:
41+
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
42+
```
43+
sam build
44+
sam deploy --guided
45+
```
4146
47+
1. During the prompts:
4248
* **Stack Name**: The name of the stack to deploy to CloudFormation. This should be unique to your account and region, and a good starting point would be something matching your project name.
4349
* **AWS Region**: The AWS region you want to deploy your app to.
44-
* **Parameter MSKClusterName**: The name of the MSKCluster
45-
* **Parameter MSKClusterId**: The unique ID of the MSKCluster
50+
* **Parameter MSKClusterName**: The name of the MSKCluster, eg. msk-test-cluster
51+
52+
* **Parameter MSKClusterId**: The unique ID of the MSKCluster, eg. a4e132c8-6ad0-4334-a313-123456789012-s2
4653
* **Parameter MSKTopic**: The Kafka topic on which the lambda function will listen on
4754
* **Confirm changes before deploy**: If set to yes, any change sets will be shown to you before execution for manual review. If set to no, the AWS SAM CLI will automatically deploy application changes.
4855
* **Allow SAM CLI IAM role creation**: Many AWS SAM templates, including this example, create AWS IAM roles required for the AWS Lambda function(s) included to access AWS services. By default, these are scoped down to minimum required permissions. To deploy an AWS CloudFormation stack which creates or modifies IAM roles, the `CAPABILITY_IAM` value for `capabilities` must be provided. If permission isn't provided through this prompt, to deploy this example you must explicitly pass `--capabilities CAPABILITY_IAM` to the `sam deploy` command.
@@ -51,29 +58,36 @@ The first command will build the source of your application. The second command
5158
* **SAM configuration file [samconfig.toml]**: Name of the configuration file to store configuration information locally
5259
* **SAM configuration environment [default]**: Environment for storing deployment information locally
5360
54-
You should get a message "Successfully created/updated stack - <StackName> in <Region>" if all goes well
61+
You should get a message "Successfully created/updated stack - <StackName> in <Region>" if all goes well.
62+
63+
Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults.
64+
65+
## How it works
66+
67+
This pattern creates a Lambda function along with a Lambda Event Source Mapping(ESM) resource. This maps a Kafka topic on an MSK Cluster as a trigger to a Lambda function. The ESM takes care of polling the Kafka topic and then invokes the Lambda function with a batch of messages.
5568
5669
## Test the sample application
5770
58-
Once the lambda function is deployed, send some Kafka messages on the topic that the lambda function is listening on, on the MSK server.
71+
Once the Lambda function is deployed, send some Kafka messages to the topic that you configured in the Lambda function trigger.
5972
6073
Either send at least 10 messages or wait for 300 seconds (check the values of BatchSize: 10 and MaximumBatchingWindowInSeconds: 300 in the template.yaml file)
6174
62-
Then check Cloudwatch logs and you should see messages for the Cloudwatch Log Group with the name of the deployed Lambda function.
75+
Then check Amazon CloudWatch logs and you should see messages in the CloudWatch Log Group with the name of the deployed Lambda function.
6376
64-
The lambda code parses the Kafka messages and outputs the fields in the Kafka messages to Cloudwatch logs
77+
The Lambda code parses the Kafka messages and outputs the fields in the Kafka messages to CloudWatch logs.
6578
66-
A single lambda function receives a batch of messages. The messages are received as a map with each key being a combination of the topic and the partition, as a single batch can receive messages from multiple partitions.
79+
A single Lambda function receives a batch of messages. The messages are received as a map with each key being a combination of the topic and the partition, as a single batch can receive messages from multiple partitions.
6780
68-
Each key has a list of messages. Each Kafka message has the following properties - Topic, Partition, Offset, TimeStamp, TimeStampType, Key and Value
81+
Each key has a list of messages. Each Kafka message has the following properties - `Topic`, `Partition`, `Offset`, `TimeStamp`, `TimeStampType`, `Key`, and `Value`.
6982
70-
The Key and Value are base64 encoded and have to be decoded. A message can also have a list of headers, each header having a key and a value.
83+
The `Key` and `Value` are base64 encoded and have to be decoded. A message can also have a list of headers, each header having a key and a value.
84+
85+
The code in this example prints out the fields in the Kafka message and also decrypts the key and the value and logs them to CloudWatch logs.
7186
72-
The code in this example prints out the fields in the Kafka message and also decrypts the key and the value and logs them in Cloudwatch logs.
7387
7488
### Local development
7589
76-
**Invoking function locally using sam local**
90+
**You can invoke the function locally using `sam local`**
7791
7892
```bash
7993
sam local invoke --event=events/event.json
@@ -106,11 +120,7 @@ REPORT RequestId: 2d1041e7-fb49-4181-a8ac-15277f5d2b6c Init Duration: 11.37 msDu
106120

107121
1. Delete the stack
108122
```bash
109-
aws cloudformation delete-stack --stack-name STACK_NAME
110-
```
111-
1. Confirm the stack has been deleted
112-
```bash
113-
aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"
123+
sam delete
114124
```
115125

116126
----

‎msk-lambda-iam-node-sam/example-pattern.json‎

Lines changed: 81 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,89 +1,90 @@
11
{
2-
"title": "Lambda function that is subscribed to an MSK Topic",
3-
"description": "Create a Lambda function that uses an Amazon MSK Topic as an event source with IAM Authentication.",
4-
"language": "Node.js",
5-
"level": "200",
6-
"framework": "SAM",
7-
"introBox": {
8-
"headline": "How it works",
9-
"text": [
10-
"This pattern provides a Lambda function along with an Event Source Mapping to a Kafka topic ",
11-
"It requires that you already have a Managed Streaming for Kafka(MSK) Cluster setup with a topic created. If you don't already have an MSK Cluster ",
12-
"You can use the example in this pattern https://serverlessland.com/patterns/msk-cfn-sasl-lambda (linked in the resources) to deploy a cluster.",
13-
"This pattern will work with either a provisioned or serverless MSK cluster as long as the cluster is configured to use IAM authentication. ",
14-
"For detailed deployment instructions instructions see the README "
15-
]
16-
},
17-
"gitHub": {
18-
"template": {
19-
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/msk-lambda-nodejs-sam",
20-
"templateURL": "serverless-patterns/msk-lambda-nodejs-sam",
21-
"projectFolder": "msk-lambda-nodejs-sam",
22-
"templateFile": "msk-lambda-nodejs-sam/template.yml"
23-
}
24-
},
25-
"resources": {
26-
"bullets": [
2+
"title": "AWS Lambda function subscribed to an Amazon MSK topic using IAM auth",
3+
"description": "Creates a Lambda function that uses an Amazon MSK topic as an event source with IAM authentication.",
4+
"language": "Node.js",
5+
"level": "200",
6+
"framework": "SAM",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern provides a Lambda function along with an Event Source Mapping to a Kafka topic.",
11+
"It requires that you already have an Amazon Managed Streaming for Kafka(Amazon MSK) cluster setup with a topic created. If you don't already have an MSK cluster ",
12+
"you can use the example in this pattern https://serverlessland.com/patterns/msk-cfn-sasl-lambda (linked in the resources) to deploy a cluster.",
13+
"This pattern works with either a Provisioned or Serverless MSK cluster as long as the cluster is configured to use IAM authentication. ",
14+
"For detailed deployment instructions instructions see the README "
15+
]
16+
},
17+
"gitHub": {
18+
"template": {
19+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/msk-lambda-node-sam",
20+
"templateURL": "serverless-patterns/msk-lambda-node-sam",
21+
"projectFolder": "msk-lambda-node-sam",
22+
"templateFile": "template.yml"
23+
}
24+
},
25+
"resources": {
26+
"bullets": [
27+
{
28+
"text": "Amazon MSK Cluster pattern",
29+
"link": "https://serverlessland.com/patterns/msk-cfn-sasl-lambda"
30+
},
31+
{
32+
"text": "Using AWS Lambda with Amazon MSK",
33+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html"
34+
},
35+
{
36+
"text": "AWS CloudFormation Provisioned MSK cluster reference",
37+
"link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-cluster.html"
38+
},
39+
{
40+
"text": "AWS CloudFormation Serverless MSK cluster reference",
41+
"link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-serverlesscluster.html"
42+
}
43+
]
44+
},
45+
"deploy": {
46+
"text": [
47+
"sam deploy --guided"
48+
]
49+
},
50+
"testing": {
51+
"text": [
52+
"See the GitHub repo for detailed testing instructions."
53+
]
54+
},
55+
"cleanup": {
56+
"text": [
57+
"Delete the template: <code>sam delete</code>."
58+
]
59+
},
60+
"authors": [
2761
{
28-
"text": "MSK Cluster Pattern",
29-
"link": "https://serverlessland.com/patterns/msk-cfn-sasl-lambda"
62+
"name": "Vaibhav Jain",
63+
"bio": "AWS - Sr. Application Architect",
64+
"image": "https://media.licdn.com/dms/image/C4E03AQEqzZWHGT4dBQ/profile-displayphoto-shrink_800_800/0/1580165399872?e=1687392000&v=beta&t=zdxENLnqCpqCz9i1Uf5Yx4YXlR9EYvgxP8N5UTsy6J8",
65+
"linkedin": "https://www.linkedin.com/in/vaibhavjainv/"
3066
},
3167
{
32-
"text": "Using Lambda with Amazon MSK",
33-
"link": "https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html"
68+
"name": "Paveen Allam",
69+
"bio": "Senior Solutions Architect",
70+
"image": "https://www.fintail.me/images/pa.jpg",
71+
"linkedin": "https://www.linkedin.com/in/pallam/"
72+
},
73+
{
74+
"name": "Suraj Tripathi",
75+
"bio": "AWS - AppDev Cloud Consultant",
76+
"linkedin": "https://www.linkedin.com/in/suraj-tripathi-01b49a140/"
3477
},
3578
{
36-
"text": "CloudFormation Provisioned MSK cluster reference",
37-
"link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-cluster.html"
79+
"name": "Adam Wagner",
80+
"bio": "AWS - Principal Serverless Solutions Architect",
81+
"linkedin": "https://www.linkedin.com/in/adam-wagner-4bb412/"
3882
},
3983
{
40-
"text": "CloudFormation serverless MSK cluster reference",
41-
"link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-serverlesscluster.html"
42-
}
84+
"name": "Indranil Banerjee",
85+
"bio": "AWS - Senior Solutions Architect",
86+
"linkedin": "https://www.linkedin.com/in/indranil-banerjee-b00a261/"
87+
}
4388
]
44-
},
45-
"deploy": {
46-
"text": [
47-
"sam deploy --guided"
48-
]
49-
},
50-
"testing": {
51-
"text": [
52-
"See the Github repo for detailed testing instructions."
53-
]
54-
},
55-
"cleanup": {
56-
"text": [
57-
"Delete the Lambda Function: <code>sam delete</code>."
58-
]
59-
},
60-
"authors": [
61-
{
62-
"name": "Vaibhav Jain",
63-
"bio": "AWS - Sr. Application Architect",
64-
"image": "https://media.licdn.com/dms/image/C4E03AQEqzZWHGT4dBQ/profile-displayphoto-shrink_800_800/0/1580165399872?e=1687392000&v=beta&t=zdxENLnqCpqCz9i1Uf5Yx4YXlR9EYvgxP8N5UTsy6J8",
65-
"linkedin": "https://www.linkedin.com/in/vaibhavjainv/"
66-
},
67-
{
68-
"name": "Paveen Allam",
69-
"bio": "Senior Solutions Architect",
70-
"image": "https://www.fintail.me/images/pa.jpg",
71-
"linkedin": "https://www.linkedin.com/in/pallam/"
72-
},
73-
{
74-
"name": "Suraj Tripathi",
75-
"bio": "AWS - AppDev Cloud Consultant",
76-
"linkedin": "https://www.linkedin.com/in/suraj-tripathi-01b49a140/"
77-
},
78-
{
79-
"name": "Adam Wagner",
80-
"bio": "AWS - Principal Serverless Solutions Architect",
81-
"linkedin": "https://www.linkedin.com/in/adam-wagner-4bb412/"
82-
},
83-
{
84-
"name": "Indranil Banerjee",
85-
"bio": "AWS - Senior Solutions Architect",
86-
"linkedin": "https://www.linkedin.com/in/indranil-banerjee-b00a261/"
87-
}
88-
]
89-
}
89+
}
90+

‎msk-lambda-iam-python-sam/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ To use the AWS SAM CLI, you need the following tools.
4040
4141
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
4242
```
43+
sam build
4344
sam deploy --guided
4445
```
4546

0 commit comments

Comments
 (0)