Skip to content

Commit 047cb6d

Browse files
Guy7BGuy7B
committed
📚 docs: improve readme
1 parent e93d293 commit 047cb6d

6 files changed

Lines changed: 153 additions & 119 deletions

File tree

‎README.md‎

Lines changed: 63 additions & 113 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,18 @@ Available as a [Serverless plugin](#serverless-plugin), [npm package](#standalon
2121
- Safe & Secure preparation data
2222
- Store preparation data in a private s3 bucket. [Prepare data for your data transformation](#usage-and-command-line-options)
2323

24+
## Table of contents
25+
26+
- [Quick Start](#quick-start)
27+
- [Serverless plugin](#⚡-Serverless-plugin)
28+
- [Standalone npm package](#standalone-npm-package)
29+
- [Interactive CLI](#💻-Interactive-CLI)
30+
- [Creating your first data transformation](#creating-your-first-data-transformation)
31+
- [Usage and command-line options](#usage-and-command-line-options)
32+
- [What happens behind the scenes](#what-happens-behind-the-scenes)
33+
- [Examples](#examples)
34+
- [The data transformation process](https://github.com/jitsecurity/dynamo-data-transform/blob/main/docs/zero_downtime_data_transformation_process.md)
35+
2436
## Quick Start
2537
### ⚡ Serverless plugin
2638
- Install
@@ -63,42 +75,35 @@ dynamodt -i
6375
![cli gif](https://user-images.githubusercontent.com/35347793/172045910-d511e735-2d31-4713-bb64-5f55a900941c.gif)
6476

6577

66-
## Table of contents
67-
68-
- [Quick Start](#quick-start)
69-
- [Usage and command-line options](#usage-and-command-line-options)
70-
- [What happens behind the scenes](#what-happens-behind-the-scenes)
71-
- [The data transformation process](#the-data-transformation-process)
72-
- [Process Steps](#steps)
73-
- [Key Concepts](#key-concepts)
74-
- [Troubleshooting](#troubleshooting)
75-
- [Examples](#examples)
76-
77-
78-
79-
## Usage and command-line options
8078

81-
List available commands:
82-
Serverless plugin:
79+
## Creating your first data transformation
80+
1. Intialize data-transformations folder
81+
Serverless (the plugin reads the table names from the serverless.yml file):
8382
```bash
84-
sls dynamodt --help
83+
sls dynamodt init
8584
```
86-
Standalone npm package:
85+
Standalone:
8786
```bash
88-
dynamodt help
87+
ddt init --tableNames <table_names>
8988
```
9089

90+
Open the generated data transformation file 'v1_script-name.js' file and implement the following functions:
91+
- transformUp: Executed when running `dynamodt up`
92+
- transformDown: Executed when running `dynamodt down -t <table>`
93+
- prepare (optional): Executed when running `dynamodt prepare -t <table> --tNumber <transformation_number>`
9194

92-
To list all of the options for a specific command run:
93-
Serverless plugin:
95+
The function parameters:
96+
- ddb: The DynamoDB Document client object see [DynamoDB Client](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb)
97+
- isDryRun: Boolean indicating if --dry run supplied. You can use it to print/log the data instead of storing it.
98+
- preparationData: if you stored the preparation data using `dynamodt prepare`, you can use it here.
99+
100+
2. Run the data transformation
94101
```bash
95-
sls dynamodt <command> --help
102+
dynamodt up
96103
```
97104

98-
## What happens behind the scenes
99-
- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
100105

101-
## Data Transformation Script Format (e.g v1_script.js)
106+
## Data Transformation Script Format
102107
```js
103108
const { utils } = require('dynamo-data-transform')
104109

@@ -128,9 +133,38 @@ module.exports = {
128133
```
129134

130135

136+
137+
## Usage and command-line options
138+
139+
List available commands:
140+
Serverless plugin:
141+
```bash
142+
sls dynamodt --help
143+
```
144+
Standalone npm package:
145+
```bash
146+
dynamodt help
147+
```
148+
149+
150+
To list all of the options for a specific command run:
151+
Serverless plugin:
152+
```bash
153+
sls dynamodt <command> --help
154+
```
155+
156+
Standalone npm package:
157+
```bash
158+
dynamodt <command> --help
159+
```
160+
161+
## What happens behind the scenes
162+
- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
163+
164+
165+
131166
## Examples
132-
Examples of data transformation code:
133-
https://github.com/jitsecurity/dynamo-data-transform/tree/main/examples/serverless-localstack/data-transformations
167+
[Examples of data transformation code](https://github.com/jitsecurity/dynamo-data-transform/tree/main/examples/serverless-localstack/data-transformations/UsersExample)
134168

135169

136170
### Insert records
@@ -143,7 +177,7 @@ const { USERS_DATA } = require('../../usersData');
143177
const TABLE_NAME = 'UsersExample';
144178

145179
/**
146-
* @param {DynamoDBDocumentClient} ddb - dynamo db document client https://github.com/aws/aws-sdk-js-v3/tree/main/clients/client-dynamodb
180+
* @param {DynamoDBDocumentClient} ddb - dynamo db document client https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb
147181
* @param {boolean} isDryRun - true if this is a dry run
148182
*/
149183
const transformUp = async ({ ddb, isDryRun }) => {
@@ -191,89 +225,5 @@ module.exports = {
191225
};
192226
```
193227

194-
### Add field using preparation data (s3 bucket)
195-
```js
196-
// Adding a new field "hasWikiPage"
197-
// "hasWikiPage" is a boolean field that is set to true if the item has a wiki page
198-
// It is calculated with a prepare function that fetches the wiki page status for each item
199-
200-
const { utils } = require('dynamo-data-transform');
201-
202-
const userAgentHeader = {
203-
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',
204-
};
205-
206-
const fetch = (...args) => import('node-fetch').then(({ default: nodeFetch }) => nodeFetch(
207-
...args,
208-
{
209-
headers: userAgentHeader,
210-
},
211-
));
212-
213-
const TABLE_NAME = 'UsersExample';
214-
215-
const transformUp = async ({ ddb, preparationData, isDryRun }) => {
216-
const addHasWikiPage = (hasWikiDict) => (item) => {
217-
const valueFromPreparation = hasWikiDict[`${item.PK}-${item.SK}`];
218-
const updatedItem = valueFromPreparation ? {
219-
...item,
220-
hasWikiPage: valueFromPreparation,
221-
} : item;
222-
return updatedItem;
223-
};
224-
225-
return utils.transformItems(
226-
ddb,
227-
TABLE_NAME,
228-
addHasWikiPage(JSON.parse(preparationData)),
229-
isDryRun,
230-
);
231-
};
232-
233-
const transformDown = async ({ ddb, isDryRun }) => {
234-
const removeHasWikiPage = (item) => {
235-
const { hasWikiPage, ...oldItem } = item;
236-
return oldItem;
237-
};
238-
239-
return utils.transformItems(ddb, TABLE_NAME, removeHasWikiPage, isDryRun);
240-
};
241-
242-
const prepare = async ({ ddb }) => {
243-
let lastEvalKey;
244-
let preparationData = {};
245-
246-
let scannedAllItems = false;
247-
248-
while (!scannedAllItems) {
249-
const { Items, LastEvaluatedKey } = await utils.getItems(ddb, lastEvalKey, TABLE_NAME);
250-
lastEvalKey = LastEvaluatedKey;
251-
252-
const currentPreparationData = await Promise.all(Items.map(async (item) => {
253-
const wikiItemUrl = `https://en.wikipedia.org/wiki/${item.name}`;
254-
const currWikiResponse = await fetch(wikiItemUrl);
255-
return {
256-
[`${item.PK}-${item.SK}`]: currWikiResponse.status === 200,
257-
};
258-
}));
259-
260-
preparationData = {
261-
...preparationData,
262-
...currentPreparationData.reduce((acc, item) => ({ ...acc, ...item }), {}),
263-
};
264-
265-
scannedAllItems = !lastEvalKey;
266-
}
267-
268-
return preparationData;
269-
};
270-
271-
module.exports = {
272-
transformUp,
273-
transformDown,
274-
prepare,
275-
transformationNumber: 4,
276-
};
277-
```
228+
For more examples of data transformation code, see the [examples](https://github.com/jitsecurity/dynamo-data-transform/tree/main/examples/serverless-localstack/data-transformations/UsersExample) folder in the repository.
278229

279-
For more examples of data transformation code, see the examples folder in the repository.

‎docs/zero_downtime_data_transformation_process.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
1-
# The safe data transformation process
1+
# Zero down time data transformation process
22
The next section describes how the data transformation process looks like, and the order of each step.
3+
4+
[Process Steps](#steps)
5+
[Troubleshooting](#troubleshooting)
6+
[Key Concepts](#key-concepts)
37
## Steps
48
### 1st Phase (Add New Resources)
59
1. Update the table resources if needed \
@@ -39,7 +43,6 @@ Implement these functions:
3943

4044

4145
### Key Concepts
42-
First of all, keep in mind that our mission is to prevent downtime while executing data transformations.
4346
- Don't override resources/data
4447
- Your code should be able to work with the old version of the data and keep it updated.
4548
- Prefer multiple data transformations over complex one.

‎examples/serverless-localstack/EXERCISE-prepare-data/v4_using_preparation_data.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
const { utils } = require('dynamo-data-transform');
66

77
const userAgentHeader = {
8-
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',
8+
'User-Agent': 'Chrome/81.0.4044.138',
99
};
1010

1111
const fetch = (...args) => import('node-fetch').then(({ default: nodeFetch }) => nodeFetch(

‎examples/serverless-localstack/data-transformations/UsersExample/v1_insert_users.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ const { USERS_DATA } = require('../../usersData');
66
const TABLE_NAME = 'UsersExample';
77

88
/**
9-
* @param {DynamoDBDocumentClient} ddb - dynamo db document client https://github.com/aws/aws-sdk-js-v3/tree/main/clients/client-dynamodb
9+
* @param {DynamoDBDocumentClient} ddb - dynamo db document client https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb
1010
* @param {boolean} isDryRun - true if this is a dry run
1111
*/
1212
const transformUp = async ({ ddb, isDryRun }) => {

‎examples/serverless-localstack/data-transformations/UsersExample/v4_using_preparation_data.js‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
// Adding a new field "hasWikiPage"
2+
// "hasWikiPage" is a boolean field that is set to true if the item has a wiki page
3+
// It is calculated with a prepare function that fetches the wiki page status for each item
4+
5+
const { utils } = require('dynamo-data-transform');
6+
7+
const userAgentHeader = {
8+
'User-Agent': 'Chrome/81.0.4044.138',
9+
};
10+
11+
const fetch = (...args) => import('node-fetch').then(({ default: nodeFetch }) => nodeFetch(
12+
...args,
13+
{
14+
headers: userAgentHeader,
15+
},
16+
));
17+
18+
const TABLE_NAME = 'UsersExample';
19+
20+
const transformUp = async ({ ddb, preparationData, isDryRun }) => {
21+
const addHasWikiPage = (hasWikiDict) => (item) => {
22+
const valueFromPreparation = hasWikiDict[`${item.PK}-${item.SK}`];
23+
const updatedItem = valueFromPreparation ? {
24+
...item,
25+
hasWikiPage: valueFromPreparation,
26+
} : item;
27+
return updatedItem;
28+
};
29+
30+
return utils.transformItems(
31+
ddb,
32+
TABLE_NAME,
33+
addHasWikiPage(JSON.parse(preparationData)),
34+
isDryRun,
35+
);
36+
};
37+
38+
const transformDown = async ({ ddb, isDryRun }) => {
39+
const removeHasWikiPage = (item) => {
40+
const { hasWikiPage, ...oldItem } = item;
41+
return oldItem;
42+
};
43+
44+
return utils.transformItems(ddb, TABLE_NAME, removeHasWikiPage, isDryRun);
45+
};
46+
47+
const prepare = async ({ ddb }) => {
48+
let lastEvalKey;
49+
let preparationData = {};
50+
51+
let scannedAllItems = false;
52+
53+
while (!scannedAllItems) {
54+
const { Items, LastEvaluatedKey } = await utils.getItems(ddb, lastEvalKey, TABLE_NAME);
55+
lastEvalKey = LastEvaluatedKey;
56+
57+
const currentPreparationData = await Promise.all(Items.map(async (item) => {
58+
const wikiItemUrl = `https://en.wikipedia.org/wiki/${item.name}`;
59+
const currWikiResponse = await fetch(wikiItemUrl);
60+
return {
61+
[`${item.PK}-${item.SK}`]: currWikiResponse.status === 200,
62+
};
63+
}));
64+
65+
preparationData = {
66+
...preparationData,
67+
...currentPreparationData.reduce((acc, item) => ({ ...acc, ...item }), {}),
68+
};
69+
70+
scannedAllItems = !lastEvalKey;
71+
}
72+
73+
return preparationData;
74+
};
75+
76+
module.exports = {
77+
transformUp,
78+
transformDown,
79+
prepare,
80+
transformationNumber: 4,
81+
};

‎src/config/transformation-template-file.js‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ const { utils } = require('dynamo-data-transform');
33
const TABLE_NAME = '{{YOUR_TABLE_NAME}}';
44

55
/**
6-
* @param {DynamoDBDocumentClient} ddb - dynamo db client of @aws-sdk https://github.com/aws/aws-sdk-js-v3/tree/main/clients/client-dynamodb
6+
* @param {DynamoDBDocumentClient} ddb - dynamo db client of @aws-sdk https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb
77
* @param {boolean} isDryRun
8-
* @returns {{transformed: number}}
8+
* @returns the number of transformed items { transformed: number }
99
*
1010
*/
1111
const transformUp = async ({ ddb, isDryRun }) => {

0 commit comments

Comments
 (0)