Skip to content

Commit 9421a6f

Browse files
committed
📚 Update readme, ✨ fix help option for cli
1 parent acc41f6 commit 9421a6f

4 files changed

Lines changed: 57 additions & 62 deletions

File tree

‎README.md‎

Lines changed: 11 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Available as a [Serverless plugin](#serverless-plugin), [npm package](#standalon
2222
- Store preparation data in a private s3 bucket. [Prepare data for your data transformation](#usage-and-command-line-options)
2323

2424
## Quick Start
25-
### Serverless plugin
25+
### âš¡ Serverless plugin
2626
- Install
2727
```bash
2828
npm install dynamo-data-transform --save-dev
@@ -34,7 +34,7 @@ plugins:
3434
```
3535
- Run
3636
```bash
37-
sls dynamodt list
37+
sls dynamodt --help
3838
```
3939

4040
### Standalone npm package
@@ -44,11 +44,11 @@ npm install -g dynamo-data-transform
4444
```
4545
- Run the tool
4646
```bash
47-
dynamodt --help
47+
dynamodt help
4848
```
4949
Or with the shortcut
5050
```bash
51-
ddt --help
51+
ddt help
5252
```
5353
### Interactive CLI
5454
After installing the npm package, run:
@@ -85,69 +85,20 @@ sls dynamodt --help
8585
```
8686
Standalone npm package:
8787
```bash
88-
dynamodt list
88+
dynamodt help
8989
```
9090

9191

9292
To list all of the options for a specific command run:
9393
Serverless plugin:
9494
```bash
95-
sls dynamotdt <command> --help
95+
sls dynamodt <command> --help
9696
```
9797

9898
## What happens behind the scenes
9999
- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
100100

101-
102-
## The safe data transformation process
103-
The next section describes how the data transformation process looks like, and the order of each step.
104-
### Steps
105-
#### 1st Phase (Add New Resources)
106-
1. Update the serverless.yml resources (if needed) \
107-
Reminder: we are not overriding existing data but creating new. [See some examples](#examples)
108-
1. Your new code should be able to write to your old and new resources which ensures that we can roll back to the previous state and prevent possible data gaps.
109-
1. Create a pull request and deploy it to every stage in your application
110-
111-
#### 2nd Phase (data transformation)
112-
113-
1. For the first time use `sls dynamodt init` it will generate a folder per table inside the root folder of your service (The name of the folder is the exact name of the table).
114-
A template data transformation file (v1.js) will be created in each table folder. \
115-
Implement these functions:
116-
1. `transformUp` - transform all of the table items to the new shape (use preparationData if needed).
117-
1. `transformDown` - transform all of the table items to the previous shape.
118-
1. `prepare` - use this function whenever your data transformation relies on data from external resources.
119-
120-
1. Export these functions and export the version of the current data transformation (set the sequence variable value. It should be the same number of the file name).
121-
122-
1. Preparing data from external resources for the data transformation can be done by using `sls dynamodt prepare`
123-
124-
Run `sls dynamodt prepare --tNumber <transformation_number> --table <table>`\
125-
The data will be stored in a S3 bucket \
126-
The data will be decrypted while running the data transformation script.
127-
128-
1. **Final Step** Create a pull request. \
129-
Note that the data transformation runs after an sls deploy command it is integrated \
130-
with lifecycle of serverless `after:deploy:deploy` hook.
131-
132-
#### 3rd Phase (Use The New Resources/Data)
133-
1. Adjust your code to work with the new data. \
134-
For example, read from the new index instead of the old one.
135-
1. Create a pull request with the updated lambdas.
136-
137-
138-
#### 4th Phase (Cleanup)
139-
1. Clean the unused data (attributes/indexes/etc).
140-
141-
142-
### Key Concepts
143-
First of all, keep in mind that our mission is to prevent downtime while executing data transformations.
144-
- Don't override resources/data
145-
- Your code should be able to work with the old version of the data and keep it updated.
146-
- To be continued...
147-
148-
149-
150-
### Data Transformation Script Format (e.g v1_script.js)
101+
## Data Transformation Script Format (e.g v1_script.js)
151102
```js
152103
const { utils } = require('dynamo-data-transform')
153104

@@ -177,12 +128,12 @@ module.exports = {
177128
```
178129

179130

180-
### Examples
131+
## Examples
181132
Examples of data transformation code:
182133
https://github.com/jitsecurity/dynamo-data-transform/tree/main/examples/serverless-localstack/data-transformations
183134

184135

185-
#### Insert records
136+
### Insert records
186137

187138
```js
188139
// Seed users data transformation
@@ -210,7 +161,7 @@ module.exports = {
210161
};
211162
```
212163

213-
#### Add a new field to each record
164+
### Add a new field to each record
214165
```js
215166
// Adding a "randomNumber" field to each item
216167
const { utils } = require('dynamo-data-transform');
@@ -241,7 +192,7 @@ module.exports = {
241192
};
242193
```
243194

244-
#### Add field using preparation data (s3 bucket)
195+
### Add field using preparation data (s3 bucket)
245196
```js
246197
// Adding a new field "hasWikiPage"
247198
// "hasWikiPage" is a boolean field that is set to true if the item has a wiki page

‎cli/index.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ const options = parseArgs(process.argv.slice(2),{
2323

2424
(() => {
2525
const [command] = options._;
26-
if(command === 'list') {
26+
if(command === 'help' || !command) {
2727
console.info('Available commands:');
2828
Object.entries(COMMAND_DESCRIPTION).forEach(([key, value]) => {
2929
console.info(` ${key} - ${value}\n`);

‎docs/images/undraw_data_extraction_re_0rd3.svg‎

Lines changed: 0 additions & 1 deletion
This file was deleted.
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# The safe data transformation process
2+
The next section describes how the data transformation process looks like, and the order of each step.
3+
## Steps
4+
### 1st Phase (Add New Resources)
5+
1. Update the table resources if needed \
6+
Reminder: we are not overriding existing data but creating new.
7+
1. Your new code should be able to write to your old and new resources which ensures that we can roll back to the previous state and prevent possible data gaps.
8+
1. Create a pull request and deploy it to every stage in your application
9+
10+
### 2nd Phase (data transformation)
11+
12+
1. For the first time use `sls dynamodt init` it will generate a folder per table inside the root folder of your service (The name of the folder is the exact name of the table).
13+
A template data transformation file (v1.js) will be created in each table folder. \
14+
Implement these functions:
15+
1. `transformUp` - transform all of the table items to the new shape (use preparationData if needed).
16+
1. `transformDown` - transform all of the table items to the previous shape.
17+
1. `prepare` - use this function whenever your data transformation relies on data from external resources.
18+
19+
1. Export these functions and export the version of the current data transformation (set the sequence variable value. It should be the same number of the file name).
20+
21+
1. Preparing data from external resources for the data transformation can be done by using `sls dynamodt prepare`
22+
23+
Run `sls dynamodt prepare --tNumber <transformation_number> --table <table>`\
24+
The data will be stored in a S3 bucket \
25+
The data will be decrypted while running the data transformation script.
26+
27+
1. **Final Step** Create a pull request. \
28+
Note that the data transformation runs after an sls deploy command it is integrated \
29+
with lifecycle of serverless `after:deploy:deploy` hook.
30+
31+
### 3rd Phase (Use The New Resources/Data)
32+
1. Adjust your code to work with the new data. \
33+
For example, read from the new index instead of the old one.
34+
1. Create a pull request with the updated lambdas.
35+
36+
37+
### 4th Phase (Cleanup)
38+
1. Clean the unused data (attributes/indexes/etc).
39+
40+
41+
### Key Concepts
42+
First of all, keep in mind that our mission is to prevent downtime while executing data transformations.
43+
- Don't override resources/data
44+
- Your code should be able to work with the old version of the data and keep it updated.
45+
- Prefer multiple data transformations over complex one.

0 commit comments

Comments
 (0)