Skip to content

Commit 3a2d5ce

Browse files
boyney123boyney123
authored
Merge pull request aws-samples#1602 from lwang-79/lwang-79-feature-sns-firehose-opensearch-cdk-typescript
2 parents e1cb555 + 5aef44f commit 3a2d5ce

11 files changed

Lines changed: 501 additions & 0 deletions

‎sns-firehose-opensearch-cdk-typescript/.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out

‎sns-firehose-opensearch-cdk-typescript/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎sns-firehose-opensearch-cdk-typescript/README.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Amazon SNS to Kinesis Firehose to OpenSearch Dashboard
2+
3+
<p align="center">
4+
<img src="./sns-firehose-opensearch.png">
5+
</p>
6+
7+
This pattern deploys infrastructures that utilize Kinesis Firehose to deliver messages sent to an SNS topic to the Amazon OpenSearch serverless collection.
8+
9+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/sns-firehose-opensearch-cdk-typescript](https://serverlessland.com/patterns/sns-firehose-opensearch-cdk-typescript)
10+
11+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
12+
13+
## Requirements
14+
15+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
16+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
17+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
18+
* [AWS CDK Installed](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html#getting_started_install)
19+
20+
## Deployment Instructions
21+
22+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
23+
``` bash
24+
git clone https://github.com/aws-samples/serverless-patterns
25+
```
26+
2. Change directory to the pattern directory:
27+
```bash
28+
cd sns-firehose-opensearch-cdk-typescript
29+
```
30+
3. Install dependencies:
31+
```bash
32+
npm install
33+
```
34+
4. Deploy the application. Before deployment, you can change the resource names in the file [sns-firehose-opensearch-cdk-typescript-stack.ts](https://github.com/aws-samples/serverless-patterns/blob/main/sns-firehose-opensearch-cdk-typescript/lib/sns-firehose-opensearch-cdk-typescript-stack.ts).
35+
```bash
36+
cdk deploy
37+
```
38+
39+
## How it works
40+
41+
Messages are sent to an SNS topic, and SNS fans out the messages to a Kinesis Data Firehose delivery stream. Firehose then streams and delivers the messages to OpenSearch, where the streaming data is loaded and indexed for analysis and searching. Additionally, failed data is backed up using an S3 bucket.
42+
43+
## Testing
44+
45+
1. Publish a message to the SNS topic either using the SNS console or by the following CLI command:
46+
```bash
47+
aws sns publish --topic-arn <topicARN> --message "Hello world"
48+
```
49+
50+
2. Open the OpenSearch Dashboard by logging in with the user configured in CDK. Dashboard url can be found from the CloudFormation stack outputs.
51+
52+
3. Click "Interact with the OpenSearch API".
53+
54+
4. Run the following query and verify that the messages appear in the query result. Please note that it may take a few minutes for the messages to be streamed and indexed.
55+
```
56+
GET _search
57+
{
58+
"query": {
59+
"match_all": {}
60+
}
61+
}
62+
```
63+
64+
## Cleanup
65+
66+
- Delete the stack
67+
```bash
68+
cdk destroy
69+
```
70+
----
71+
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
72+
73+
SPDX-License-Identifier: MIT-0

‎sns-firehose-opensearch-cdk-typescript/bin/sns-firehose-opensearch-cdk-typescript.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from 'aws-cdk-lib';
4+
import { SnsFirehoseOpensearchCdkTypescriptStack } from '../lib/sns-firehose-opensearch-cdk-typescript-stack';
5+
6+
const app = new cdk.App();
7+
new SnsFirehoseOpensearchCdkTypescriptStack(app, 'SnsFirehoseOpensearchCdkTypescriptStack', {
8+
/* If you don't specify 'env', this stack will be environment-agnostic.
9+
* Account/Region-dependent features and context lookups will not work,
10+
* but a single synthesized template can be deployed anywhere. */
11+
12+
/* Uncomment the next line to specialize this stack for the AWS Account
13+
* and Region that are implied by the current CLI configuration. */
14+
// env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
15+
16+
/* Uncomment the next line if you know exactly what Account and Region you
17+
* want to deploy the stack to. */
18+
// env: { account: '123456789012', region: 'us-east-1' },
19+
20+
/* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
21+
});

‎sns-firehose-opensearch-cdk-typescript/cdk.json‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/sns-firehose-opensearch-cdk-typescript.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-iam:standardizedServicePrincipals": true,
38+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
39+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
40+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
41+
"@aws-cdk/aws-route53-patters:useCertificate": true,
42+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
43+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
44+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
45+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
46+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
47+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
48+
"@aws-cdk/aws-redshift:columnId": true,
49+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
50+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
51+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
52+
"@aws-cdk/aws-kms:aliasNameRef": true,
53+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
54+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
55+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true
56+
}
57+
}

‎sns-firehose-opensearch-cdk-typescript/example-pattern.json‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
{
2+
"title": "Amazon SNS to Kinesis Firehose to OpenSearch Dashboard",
3+
"description": "This pattern deploys infrastructure that utilizes Kinesis Firehose to deliver messages sent to an SNS topic to Amazon OpenSearch.,
4+
"language": "TypeScript",
5+
"level": "200",
6+
"framework": "CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"Messages are sent to an SNS topic, and SNS fans out the messages to a Kinesis Data Firehose delivery stream. Firehose then streams and delivers the messages to OpenSearch, where the streaming data is loaded and indexed for analysis and searching. Additionally, failed data is backed up using an S3 bucket."
11+
]
12+
},
13+
"gitHub": {
14+
"template": {
15+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/sns-firehose-opensearch-cdk-typescript",
16+
"templateURL": "serverless-patterns/sns-firehose-opensearch-cdk-typescript",
17+
"projectFolder": "sns-firehose-opensearch-cdk-typescript",
18+
"templateFile": "lib/sns-firehose-opensearch-cdk-typescript-stack.ts"
19+
}
20+
},
21+
"resources": {
22+
"bullets": [
23+
{
24+
"text": "Fanout to Kinesis Data Firehose delivery streams",
25+
"link": "https://docs.amazonaws.cn/en_us/sns/latest/dg/sns-firehose-as-subscriber.html"
26+
},
27+
{
28+
"text": "Introducing message archiving and analytics for Amazon SNS",
29+
"link": "https://aws.amazon.com/blogs/compute/introducing-message-archiving-and-analytics-for-amazon-sns/"
30+
}
31+
]
32+
},
33+
"deploy": {
34+
"text": [
35+
"cdk deploy"
36+
]
37+
},
38+
"testing": {
39+
"text": [
40+
"See the GitHub repo for detailed testing instructions."
41+
]
42+
},
43+
"cleanup": {
44+
"text": [
45+
"cdk destroy"
46+
]
47+
},
48+
"authors": [
49+
{
50+
"name": "Leon Wang",
51+
"image": "https://drive.google.com/file/d/1cPkmsCOQBFdmv5_70AfyDvTDxBRX-doQ/view?usp=sharing",
52+
"bio": "I am a Cloud Support Engineer at AWS who loves coding",
53+
"linkedin": "https://www.linkedin.com/in/leon-wang-23a6988a/"
54+
}
55+
]
56+
}

‎sns-firehose-opensearch-cdk-typescript/jest.config.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
module.exports = {
2+
testEnvironment: 'node',
3+
roots: ['<rootDir>/test'],
4+
testMatch: ['**/*.test.ts'],
5+
transform: {
6+
'^.+\\.tsx?$': 'ts-jest'
7+
}
8+
};

0 commit comments

Comments
 (0)