Skip to content

Commit f08fa21

Browse files
singledigitsingledigit
authored
Merge pull request aws-samples#1601 from lefkarag/lefkarag-feature-kinesis-lambda-cdk-java
New serverless pattern - Kinesis Data Streams integration with Lambda using the AWS CDK in Java
2 parents ac048d1 + d5615d2 commit f08fa21

10 files changed

Lines changed: 459 additions & 0 deletions

File tree

‎kinesis-lambda-cdk-java/.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
*.swp
2+
package-lock.json
3+
__pycache__
4+
.pytest_cache
5+
.env
6+
.venv
7+
*.egg-info
8+
.idea
9+
10+
# CDK asset staging directory
11+
.cdk.staging
12+
cdk.out
13+
/.idea/

‎kinesis-lambda-cdk-java/README.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Amazon API Gateway Canary Deployment
2+
3+
This pattern creates an Amazon [Kinesis Data Stream](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) and an AWS [Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), using the AWS Cloud Development Kit (AWS CDK) in Java.
4+
5+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/kinesis-lambda-cdk-java](https://serverlessland.com/patterns/kinesis-lambda-cdk-java).
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
## Requirements
10+
11+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
12+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
13+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
14+
* [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) installed and configured
15+
* [Java 11+](https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html) installed
16+
* [Docker](https://docs.docker.com/get-docker/) Installed
17+
## Deployment Instructions
18+
19+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
20+
```
21+
git clone https://github.com/aws-samples/serverless-patterns
22+
```
23+
2. Change directory to the pattern directory:
24+
```
25+
cd serverless-patterns/kinesis-lambda-cdk-java
26+
```
27+
3. From the command line, use AWS CDK to deploy the AWS resources for the serverless application as specified in the MyServerlessApplicationStack.java file:
28+
```
29+
cdk deploy
30+
```
31+
32+
## How it works
33+
34+
When data is sent to the Kinesis data stream, the Lambda function will be triggered and process data from the stream.
35+
36+
## Testing
37+
38+
From the command line, run the following command to send a single data record to the Kinesis data stream. Note that you must edit the {MyServerlessApplicationStack.KinesisLambda-KinesisStream} with the Kinesis data stream ARN that is deployed. This is provided in the MyServerlessApplicationStack deployment outputs.
39+
40+
```
41+
aws kinesis put-record --stream-arn {MyServerlessApplicationStack.KinesisLambda-KinesisStream} --partition-key 123 --data testdata
42+
```
43+
44+
## Cleanup
45+
46+
1. From the command line the AWS CDK to delete the Serverless application stack
47+
```
48+
cdk destroy
49+
```
50+
----
51+
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
52+
53+
SPDX-License-Identifier: MIT-0

‎kinesis-lambda-cdk-java/cdk.json‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
{
2+
"app": "mvn -e -q compile exec:java",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"target",
11+
"pom.xml",
12+
"src/test"
13+
]
14+
},
15+
"context": {
16+
"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
17+
"@aws-cdk/core:stackRelativeExports": true,
18+
"@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
19+
"@aws-cdk/aws-lambda:recognizeVersionProps": true,
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
22+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
23+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
24+
"@aws-cdk/core:checkSecretUsage": true,
25+
"@aws-cdk/aws-iam:minimizePolicies": true,
26+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
27+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
28+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
29+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
30+
"@aws-cdk/core:target-partitions": [
31+
"aws",
32+
"aws-cn"
33+
]
34+
}
35+
}

‎kinesis-lambda-cdk-java/example-pattern.json‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
{
2+
"title": "Amazon Kinesis Data Streams triggers AWS Lambda",
3+
"description": "Create a serverless app using Amazon Kinesis Data Streams and AWS Lambda.",
4+
"language": "Java",
5+
"level": "200",
6+
"framework": "CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern creates an Amazon Kinesis Data Stream that integrates with AWS Lambda using the AWS CDK in Java.",
11+
"In this example, the Kinesis data stream is used as an event source for the AWS Lambda (Java) function which processes the data from the stream and catches any failures."
12+
]
13+
},
14+
"gitHub": {
15+
"template": {
16+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/kinesis-lambda-cdk-java",
17+
"templateURL": "patterns/kinesis-lambda-cdk-java",
18+
"projectFolder": "kinesis-lambda-cdk-java",
19+
"templateFile": "kinesis-lambda-cdk-java/src/main/java/com.myorg/MyServerlessApplicationStack.java"
20+
}
21+
},
22+
"resources": {
23+
"bullets": [
24+
{
25+
"text": "Using AWS Lambda with Amazon Kinesis",
26+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html"
27+
},
28+
{
29+
"text": "Working with the AWS CDK in Java",
30+
"link": "https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-java.html"
31+
}
32+
]
33+
},
34+
"deploy": {
35+
"text": [
36+
"cdk deploy"
37+
]
38+
},
39+
"testing": {
40+
"text": [
41+
"See the GitHub repo for detailed testing instructions."
42+
]
43+
},
44+
"cleanup": {
45+
"text": [
46+
"Delete the stack: <code>cdk destroy</code>."
47+
]
48+
},
49+
"authors": [
50+
{
51+
"name": "Lefteris Karageorgiou",
52+
"image": "https://media.licdn.com/dms/image/C4D03AQHjv8OCPlhz6w/profile-displayphoto-shrink_400_400/0/1517428865562?e=1698278400&v=beta&t=2572XBtbjoiroPWBk6K6mv3gXdMoD6IR9dwXMk83DtA",
53+
"bio": "Solutions Architect @ AWS",
54+
"linkedin": "https://www.linkedin.com/in/lefteris-karageorgiou-ba1ab926/"
55+
}
56+
]
57+
}

‎kinesis-lambda-cdk-java/package.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"dependencies": {
3+
"aws-cdk": "2.80.0"
4+
}
5+
}

‎kinesis-lambda-cdk-java/pom.xml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
3+
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
4+
<modelVersion>4.0.0</modelVersion>
5+
6+
<groupId>com.myorg</groupId>
7+
<artifactId>kinesis-lambda-cdk-java</artifactId>
8+
<version>0.1</version>
9+
10+
<properties>
11+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
12+
<cdk.version>2.32.1</cdk.version>
13+
<constructs.version>[10.0.0,11.0.0)</constructs.version>
14+
<junit.version>5.7.1</junit.version>
15+
</properties>
16+
17+
<build>
18+
<plugins>
19+
<plugin>
20+
<groupId>org.apache.maven.plugins</groupId>
21+
<artifactId>maven-compiler-plugin</artifactId>
22+
<version>3.8.1</version>
23+
<configuration>
24+
<source>11</source>
25+
<target>11</target>
26+
</configuration>
27+
</plugin>
28+
29+
<plugin>
30+
<groupId>org.codehaus.mojo</groupId>
31+
<artifactId>exec-maven-plugin</artifactId>
32+
<version>3.0.0</version>
33+
<configuration>
34+
<mainClass>com.myorg.MyServerlessApplicationApp</mainClass>
35+
</configuration>
36+
</plugin>
37+
</plugins>
38+
</build>
39+
40+
<dependencies>
41+
<!-- AWS Cloud Development Kit -->
42+
<dependency>
43+
<groupId>software.amazon.awscdk</groupId>
44+
<artifactId>aws-cdk-lib</artifactId>
45+
<version>${cdk.version}</version>
46+
</dependency>
47+
48+
49+
<dependency>
50+
<groupId>software.constructs</groupId>
51+
<artifactId>constructs</artifactId>
52+
<version>${constructs.version}</version>
53+
</dependency>
54+
55+
<dependency>
56+
<groupId>org.junit.jupiter</groupId>
57+
<artifactId>junit-jupiter</artifactId>
58+
<version>${junit.version}</version>
59+
<scope>test</scope>
60+
</dependency>
61+
</dependencies>
62+
</project>

‎kinesis-lambda-cdk-java/software/LambdaFunction/pom.xml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<groupId>cdk-sample</groupId>
5+
<artifactId>kinesis-lambda-java</artifactId>
6+
<version>1.0</version>
7+
<packaging>jar</packaging>
8+
<name>MyFunction</name>
9+
<properties>
10+
<maven.compiler.source>11</maven.compiler.source>
11+
<maven.compiler.target>11</maven.compiler.target>
12+
<log4j.version>2.18.0</log4j.version>
13+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
14+
</properties>
15+
16+
<dependencies>
17+
<dependency>
18+
<groupId>software.amazon.lambda</groupId>
19+
<artifactId>powertools-tracing</artifactId>
20+
<version>1.12.3</version>
21+
</dependency>
22+
<dependency>
23+
<groupId>software.amazon.lambda</groupId>
24+
<artifactId>powertools-metrics</artifactId>
25+
<version>1.12.3</version>
26+
</dependency>
27+
<dependency>
28+
<groupId>com.amazonaws</groupId>
29+
<artifactId>aws-lambda-java-core</artifactId>
30+
<version>1.2.1</version>
31+
</dependency>
32+
<dependency>
33+
<groupId>com.amazonaws</groupId>
34+
<artifactId>aws-lambda-java-events</artifactId>
35+
<version>3.11.0</version>
36+
</dependency>
37+
<dependency>
38+
<groupId>org.apache.logging.log4j</groupId>
39+
<artifactId>log4j-core</artifactId>
40+
<version>${log4j.version}</version>
41+
</dependency>
42+
<dependency>
43+
<groupId>org.apache.logging.log4j</groupId>
44+
<artifactId>log4j-api</artifactId>
45+
<version>${log4j.version}</version>
46+
</dependency>
47+
48+
<dependency>
49+
<groupId>junit</groupId>
50+
<artifactId>junit</artifactId>
51+
<version>4.13.2</version>
52+
<scope>test</scope>
53+
</dependency>
54+
</dependencies>
55+
56+
<build>
57+
<plugins>
58+
<plugin>
59+
<groupId>org.codehaus.mojo</groupId>
60+
<artifactId>aspectj-maven-plugin</artifactId>
61+
<version>1.14.0</version>
62+
<configuration>
63+
<source>${maven.compiler.source}</source>
64+
<target>${maven.compiler.target}</target>
65+
<complianceLevel>${maven.compiler.target}</complianceLevel>
66+
<aspectLibraries>
67+
<aspectLibrary>
68+
<groupId>software.amazon.lambda</groupId>
69+
<artifactId>powertools-tracing</artifactId>
70+
</aspectLibrary>
71+
<aspectLibrary>
72+
<groupId>software.amazon.lambda</groupId>
73+
<artifactId>powertools-metrics</artifactId>
74+
</aspectLibrary>
75+
</aspectLibraries>
76+
</configuration>
77+
<executions>
78+
<execution>
79+
<goals>
80+
<goal>compile</goal>
81+
</goals>
82+
</execution>
83+
</executions>
84+
</plugin>
85+
<plugin>
86+
<groupId>org.apache.maven.plugins</groupId>
87+
<artifactId>maven-shade-plugin</artifactId>
88+
<version>3.3.0</version>
89+
<configuration>
90+
<createDependencyReducedPom>false</createDependencyReducedPom>
91+
<finalName>myfunction</finalName>
92+
</configuration>
93+
<executions>
94+
<execution>
95+
<phase>package</phase>
96+
<goals>
97+
<goal>shade</goal>
98+
</goals>
99+
</execution>
100+
</executions>
101+
</plugin>
102+
</plugins>
103+
</build>
104+
</project>

‎kinesis-lambda-cdk-java/software/LambdaFunction/src/main/java/com/example/MyFunction.java‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
package com.example;
2+
3+
import com.amazonaws.services.lambda.runtime.Context;
4+
import com.amazonaws.services.lambda.runtime.RequestHandler;
5+
import com.amazonaws.services.lambda.runtime.events.KinesisEvent;
6+
import com.amazonaws.services.lambda.runtime.events.StreamsEventResponse;
7+
8+
import java.io.Serializable;
9+
import java.util.ArrayList;
10+
import java.util.List;
11+
12+
import org.slf4j.Logger;
13+
import org.slf4j.LoggerFactory;
14+
15+
public class MyFunction implements RequestHandler<KinesisEvent, StreamsEventResponse> {
16+
17+
static Logger logger = LoggerFactory.getLogger(MyFunction.class);
18+
19+
@Override
20+
public StreamsEventResponse handleRequest(KinesisEvent input, Context context) {
21+
22+
logger.info("Starting");
23+
24+
List<StreamsEventResponse.BatchItemFailure> batchItemFailures = new ArrayList<>();
25+
String curRecordSequenceNumber = "";
26+
27+
for (KinesisEvent.KinesisEventRecord kinesisEventRecord : input.getRecords()) {
28+
try {
29+
//Process your record
30+
KinesisEvent.Record kinesisRecord = kinesisEventRecord.getKinesis();
31+
curRecordSequenceNumber = kinesisRecord.getSequenceNumber();
32+
33+
} catch (Exception e) {
34+
/* Since we are working with streams, we can return the failed item immediately.
35+
Lambda will immediately begin to retry processing from this failed item onwards. */
36+
batchItemFailures.add(new StreamsEventResponse.BatchItemFailure(curRecordSequenceNumber));
37+
return new StreamsEventResponse(batchItemFailures);
38+
}
39+
}
40+
41+
return new StreamsEventResponse(batchItemFailures);
42+
}
43+
}

‎kinesis-lambda-cdk-java/src/main/java/com/myorg/MyServerlessApplicationApp.java‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
package com.myorg;
2+
3+
import software.amazon.awscdk.App;
4+
import software.amazon.awscdk.StackProps;
5+
6+
7+
public class MyServerlessApplicationApp {
8+
public static void main(final String[] args) {
9+
App app = new App();
10+
11+
new MyServerlessApplicationStack(app, "MyServerlessApplicationStack", StackProps.builder()
12+
.build());
13+
14+
app.synth();
15+
}
16+
}
17+

0 commit comments

Comments
 (0)