Skip to content

Commit 16f3022

Browse files
srisub-amznsrisub-amzn
committed
Populate repository
1 parent 04c998b commit 16f3022

5 files changed

Lines changed: 421 additions & 2 deletions

File tree

‎LICENSE.txt‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
Amazon Software License
2+
3+
This Amazon Software License (“License”) governs your use, reproduction, and distribution of the accompanying software as specified below.
4+
5+
1. Definitions
6+
“Licensor” means any person or entity that distributes its Work.
7+
8+
“Software” means the original work of authorship made available under this License.
9+
10+
“Work” means the Software and any additions to or derivative works of the Software that are made available under this License.
11+
12+
The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this License, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work.
13+
14+
Works, including the Software, are “made available” under this License by including in or with the Work either (a) a copyright notice referencing the applicability of this License to the Work, or (b) a copy of this License.
15+
16+
2. License Grants
17+
2.1 Copyright Grant. Subject to the terms and conditions of this License, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form.
18+
2.2 Patent Grant. Subject to the terms and conditions of this License, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free patent license to make, have made, use, sell, offer for sale, import, and otherwise transfer its Work, in whole or in part. The foregoing license applies only to the patent claims licensable by Licensor that would be infringed by Licensor’s Work (or portion thereof) individually and excluding any combinations with any other materials or technology.
19+
20+
3. Limitations
21+
3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this License, (b) you include a complete copy of this License with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work.
22+
3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this License (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself.
23+
3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use with the web services, computing platforms or applications provided by Amazon.com, Inc. or its affiliates, including Amazon Web Services, Inc.
24+
3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this License from such Licensor (including the grants in Sections 2.1 and 2.2) will terminate immediately.
25+
3.5 Trademarks. This License does not grant any rights to use any Licensor’s or its affiliates’ names, logos, or trademarks, except as necessary to reproduce the notices described in this License.
26+
3.6 Termination. If you violate any term of this License, then your rights under this License (including the grants in Sections 2.1 and 2.2) will terminate immediately.
27+
28+
4. Disclaimer of Warranty.
29+
THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF M ERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE. SOME STATES’ CONSUMER LAWS DO NOT ALLOW EXCLUSION OF AN IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU.
30+
31+
5. Limitation of Liability.
32+
EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER COMM ERCIAL DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
33+
Effective Date – April 18, 2008 © 2008 Amazon.com, Inc. or its affiliates. All rights reserved.

‎NOTICE.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
amazon-elasticsearch-lambda-samples
2+
Copyright 2015- Amazon.com, Inc. or its affiliates. All Rights Reserved.

‎README.md‎

Lines changed: 168 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,168 @@
1-
# amazon-elasticsearch-lambda-samples
2-
Data ingestion for Amazon Elasticsearch Service from S3 and Amazon Kinesis, using AWS Lambda: Sample code
1+
# Streaming Data to Amazon Elasticsearch Service
2+
## Using AWS Lambda: Sample Node.js Code
3+
### Package amazon-elasticsearch-lambda-samples
4+
5+
Copyright 2015- Amazon.com, Inc. or its affiliates. All Rights Reserved.
6+
7+
### Introduction
8+
It is often useful to stream data, as it gets generated, for indexing in an
9+
Amazon Elasticsearch Service domain. This helps fresh data to be available for
10+
search or analytics. To do this requires:
11+
12+
1. Knowing when new data is available
13+
2. Code to pick up and parse the data into JSON documents, and add them to an
14+
Amazon Elasticsearch (henceforth, ES for short) domain.
15+
3. Scalable and fully managed infrastructure to host this code
16+
17+
*Lambda* is an AWS service that takes care of these requirements. Put simply,
18+
it is an "event handling" service in the cloud. Lambda lets us implement
19+
the event handler (in Node.js or Java), which it hosts and invokes in response
20+
to an event.
21+
22+
The handler can be triggered by a "push" or a "pull" approach.
23+
Certain event sources (such as S3) push an event notification to Lambda.
24+
Others (such as Kinesis) require Lambda to poll for events and pull them
25+
when available.
26+
27+
For more details on AWS Lambda, please see
28+
[the documentation](http://aws.amazon.com/documentation/lambda/).
29+
30+
This package contains sample Lambda code (in Node.js) to stream data to ES
31+
from two common AWS data sources: S3 and Kinesis. The S3 sample takes apache
32+
log files, parses them into JSON documents and adds them to ES. The Kinesis
33+
sample reads JSON data from the stream and adds them to ES.
34+
35+
Note that the sample code has been kept simple for reasons for clarity. It
36+
does not handle ES document batching, or eventual consistency issues for
37+
S3 updates, etc.
38+
39+
### Setup Overview
40+
41+
While some detailed instructions are covered later in this file and elsewhere
42+
(in the Lambda documentation), this section aims to show the larger picture
43+
that the individual steps work to accomplish. We assume that the data source
44+
(an S3 bucket or a Kinesis stream, in this case) and an ES domain are already
45+
set up.
46+
47+
1. **Deployment Package**: The "Deployment Package" is the event handler code files
48+
and its dependencies packaged as a zip file. The first step in creating
49+
a new Lambda function is to prepare and upload this zip file.
50+
51+
2. **Lambda Configuration**:
52+
53+
1. Handler: The name of the main code file in the deployment package,
54+
with the file extension replaced with a `.handler` suffix.
55+
2. Memory: The memory limit, based on which the EC2 instance type to use
56+
is determined. For now, the default should do.
57+
3. Timeout: The default timeout value (3 seconds) is quite low for our
58+
use-case. 10 seconds might work better, but please adjust based on
59+
your testing.
60+
61+
3. **Authorization**: Since there is a need here for various AWS services making
62+
calls to each other, appropriate authorization is required. This takes
63+
the form of configuring an IAM role, to which various authorization policies
64+
are attached. This role will be assumed by the Lambda function when running.
65+
66+
Note:
67+
68+
* The AWS Console is simpler to use for configuration than other methods.
69+
* Lambda is currently available only in a few regions (us-east-1, us-west-2,
70+
eu-west-1, ap-northeast-1).
71+
* Once the setup is complete and tested, enable the data source in the Lambda
72+
console, so that data may start streaming in.
73+
* The code is kept simple for purposes of illustration. It doesn't batch
74+
documents when loading the ES domain, or (for S3 updates) handle
75+
eventual consistency cases.
76+
77+
#### Deployment Package Creation
78+
1. On your development machine, download and install [Node.js](https://nodejs.org/en/).
79+
2. Anywhere, create a directory structure similar to the following:
80+
81+
eslambda (place sample code here)
82+
|
83+
+-- node_modules (dependencies will go here)
84+
85+
3. Modify the sample code with the correct ES endpoint, region, index
86+
and document type.
87+
4. Install each dependency imported by the sample code
88+
(with the `require()` call), as follows:
89+
90+
npm install <dependency>
91+
92+
Verify that these are installed within the `node_modules` subdirectory.
93+
5. Create a zip file to package the code and the `node_modules` subdirectory
94+
95+
zip -r eslambda.zip *
96+
97+
The zip file thus created is the Lambda Deployment Package.
98+
99+
## S3-Lambda-ES
100+
101+
Set up the Lambda function and the S3 bucket as described in the
102+
[Lambda-S3 Walkthrough](http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser.html).
103+
Please keep in mind the following notes and configuration overrides:
104+
105+
* The walkthrough uses the AWS CLI for configuration, but it's probably more
106+
convenient to use the AWS Console (web UI)
107+
108+
* The S3 bucket must be created in the same region as Lambda is, so that it
109+
can push events to Lambda.
110+
111+
* When registering the S3 bucket as the data-source in Lambda, add a filter
112+
for files having `.log` suffix, so that Lambda picks up only apache log files.
113+
114+
* The following authorizations are required:
115+
116+
1. Lambda permits S3 to push event notification to it
117+
2. S3 permits Lambda to fetch the created objects from a given bucket
118+
3. ES permits Lambda to add documents to the given domain
119+
120+
The Lambda console provides a simple way to create an IAM role with policies
121+
for (1). For (2), when creating the IAM role, choose the "S3 execution role"
122+
option; this will load the role with permissions to read from the S3
123+
bucket. For (3), add the following access policy to permit ES operations
124+
to the role.
125+
126+
{
127+
"Version": "2012-10-17",
128+
"Statement": [
129+
{
130+
"Action": [
131+
"es:*"
132+
],
133+
"Effect": "Allow",
134+
"Resource": "*"
135+
}
136+
]
137+
}
138+
139+
140+
## Kinesis-Lambda-ES
141+
142+
Set up the Lambda function and the Kinesis stream as described in the
143+
[Lambda-Kinesis Walkthrough](http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-kinesis-events-adminuser.html).
144+
Please keep in mind the following notes and configuration overrides:
145+
146+
* The walkthrough uses the AWS CLI, but it's probably more convenient to use
147+
the AWS Console (web UI) for Lambda configuration.
148+
149+
* To the IAM role assigned to the Lambda function, add the following
150+
access policy to permit ES operations.
151+
152+
{
153+
"Version": "2012-10-17",
154+
"Statement": [
155+
{
156+
"Action": [
157+
"es:*"
158+
],
159+
"Effect": "Allow",
160+
"Resource": "*"
161+
}
162+
]
163+
}
164+
165+
* For testing: If you have a Kinesis client, use it to stream a record to Lambda.
166+
If not, the AWS CLI could be used to push a JSON document to Lambda.
167+
168+
aws kinesis put-record --stream-name <lambda name> --data "<JSON document>" --region <region> --partition-key shardId-000000000000

‎src/kinesis_lambda_es.js‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
/*
2+
* Sample node.js code for AWS Lambda to upload the JSON documents
3+
* pushed from Kinesis to Amazon Elasticsearch.
4+
*
5+
*
6+
* Copyright 2015- Amazon.com, Inc. or its affiliates. All Rights Reserved.
7+
*
8+
* Licensed under the Amazon Software License (the "License").
9+
* You may not use this file except in compliance with the License.
10+
* A copy of the License is located at http://aws.amazon.com/asl/
11+
* or in the "license" file accompanying this file. This file is distributed
12+
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
13+
* express or implied. See the License for the specific language governing
14+
* permissions and limitations under the License.
15+
*/
16+
17+
/* == Imports == */
18+
var AWS = require('aws-sdk');
19+
var path = require('path');
20+
21+
/* == Globals == */
22+
var esDomain = {
23+
region: 'us-east-1',
24+
endpoint: 'my-domain-search-endpoint',
25+
index: 'myindex',
26+
doctype: 'mytype'
27+
};
28+
var endpoint = new AWS.Endpoint(esDomain.endpoint);
29+
/*
30+
* The AWS credentials are picked up from the environment.
31+
* They belong to the IAM role assigned to the Lambda function.
32+
* Since the ES requests are signed using these credentials,
33+
* make sure to apply a policy that allows ES domain operations
34+
* to the role.
35+
*/
36+
var creds = new AWS.EnvironmentCredentials('AWS');
37+
38+
39+
/* Lambda "main": Execution begins here */
40+
exports.handler = function(event, context) {
41+
console.log(JSON.stringify(event, null, ' '));
42+
event.Records.forEach(function(record) {
43+
var jsonDoc = new Buffer(record.kinesis.data, 'base64');
44+
postToES(jsonDoc.toString(), context);
45+
});
46+
}
47+
48+
49+
/*
50+
* Post the given document to Elasticsearch
51+
*/
52+
function postToES(doc, context) {
53+
var req = new AWS.HttpRequest(endpoint);
54+
55+
req.method = 'POST';
56+
req.path = path.join('/', esDomain.index, esDomain.doctype);
57+
req.region = esDomain.region;
58+
req.headers['presigned-expires'] = false;
59+
req.headers['Host'] = endpoint.host;
60+
req.body = doc;
61+
62+
var signer = new AWS.Signers.V4(req , 'es'); // es: service code
63+
signer.addAuthorization(creds, new Date());
64+
65+
var send = new AWS.NodeHttpClient();
66+
send.handleRequest(req, null, function(httpResp) {
67+
var respBody = '';
68+
httpResp.on('data', function (chunk) {
69+
respBody += chunk;
70+
});
71+
httpResp.on('end', function (chunk) {
72+
console.log('Response: ' + respBody);
73+
context.succeed('Lambda added document ' + doc);
74+
});
75+
}, function(err) {
76+
console.log('Error: ' + err);
77+
context.fail('Lambda failed with error ' + err);
78+
});
79+
}

0 commit comments

Comments
 (0)