Skip to content

Commit ea5c974

Browse files
committed
📚 update readme
1 parent 3daf948 commit ea5c974

2 files changed

Lines changed: 47 additions & 68 deletions

File tree

README.md

Lines changed: 40 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,30 @@
1-
# DynamoDB data transformations Tool
1+
![ddt_graphic1x](https://user-images.githubusercontent.com/101042972/171440725-b9e7bad8-11d8-4202-965a-96d1f5d9b2b1.jpg)
2+
23

34
<p>
45
<a href="https://www.serverless.com">
56
<img src="http://public.serverless.com/badges/v3.svg">
67
</a>
78
</p>
89

9-
## Introduction:
10-
- We design our databases schemas and define how we want to store the data inside. \
11-
We hope that our design will last forever, \
12-
however, in the real world, it might not happen. \
13-
Making changes in our database design is a regular process that happens sometimes.
10+
Dynamo Data Transform is an easy to use data transformation tool for DynamoDB.
11+
12+
It allows performing powerful data transformations using simple Javascript commands, without the risk of breaking your database.
13+
Available as a [Serverless plugin](#serverless-plugin), [npm package](#standalone-npm-package) and even as an [interactive CLI](#interactive-cli), Dynamo Data Transform saves you time and keeps you safe with features like dry-running a data transformation and even rolling back your last trasnformation if needed.
1414

15-
- We need to be able to execute data transformations \
16-
without interfering with us or our clients \
17-
and without losing the ability to get back to the previous state of the data.
15+
![cli-image](https://user-images.githubusercontent.com/101042972/171456161-fe2437b7-06be-4106-a0e0-acbf7d442a60.JPG)
16+
17+
**Features**
1818

19-
![:)](./docs/images/undraw_data_extraction_re_0rd3.svg)
19+
- Seemless data transformations management.
20+
- Support for multiple stages.
21+
- History of executed data transformations.
22+
- Dry run option for each command (by suppling --dry flag, the data will be printed instead of stored).
23+
- Safe & Secure preparation data
24+
- Store preparation data in a private s3 bucket. [Prepare data for your data transformation](#usage-and-command-line-options)
2025

21-
## Quick Start:
22-
### Serverless plugin:
26+
## Quick Start
27+
### Serverless plugin
2328
- Install
2429
```bash
2530
npm install dynamo-data-transform --save-dev
@@ -34,7 +39,7 @@ plugins:
3439
sls dynamodt list
3540
```
3641

37-
### Standalone npm package:
42+
### Standalone npm package
3843
- Install the tool
3944
```bash
4045
npm install -g dynamo-data-transform
@@ -47,37 +52,26 @@ Or with the shortcut
4752
```bash
4853
ddt --help
4954
```
50-
- Use the interactive cli
55+
### Interactive CLI
56+
After installing the npm package, run:
5157
```bash
5258
dynamodt -i
5359
```
5460

5561

5662

5763

58-
**Features**
5964

60-
- Seemless data transformations management.
61-
- Support for multiple stages.
62-
- History of executed data transformations.
63-
- Dry run option for each command (by suppling --dry flag, the data will be printed instead of stored).
64-
- Safe & Secure preparation data - \
65-
Store preparation data in a private s3 bucket. \
66-
[Prepare data for your data transformation](#usage-and-command-line-options)
6765

6866

6967

70-
## Documentation
68+
## Table of contents
7169

72-
- [Introduction:](#introduction)
73-
- [Documentation](#documentation)
74-
- [Installation](#installation)
70+
- [Quick Start](#quick-start)
7571
- [Usage and command-line options](#usage-and-command-line-options)
76-
- [What it does behind the scenes:](#what-it-does-behind-the-scenes)
77-
- [The data transformation process:](#the-data-transformation-process)
78-
- [Steps](#steps)
79-
- [First Phase:](#first-phase)
80-
- [The Second Phase (data transformation):](#the-second-phase-data-transformation)
72+
- [What happens behind the scenes](#what-happens-behind-the-scenes)
73+
- [The data transformation process](#the-data-transformation-process)
74+
- [Process Steps](#steps)
8175
- [Key Concepts](#key-concepts)
8276
- [Troubleshooting](#troubleshooting)
8377
- [Examples](#examples)
@@ -103,21 +97,20 @@ Serverless plugin:
10397
sls dynamotdt <command> --help
10498
```
10599

106-
## What it does behind the scenes
107-
- When a data transformation is running for the first time a Record in your table is created. \
108-
This record is for tracking the executed transformations on a specific table.
100+
## What happens behind the scenes
101+
- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
109102

110103

111104
## The safe data transformation process
112-
The next section describes how the process looks like and the order of every step in the data transformation process.
105+
The next section describes how the data transformation process looks like, and the order of each step.
113106
### Steps
114-
#### First Phase (Add New Resources)
107+
#### 1st Phase (Add New Resources)
115108
1. Update the serverless.yml resources (if needed) \
116109
Reminder: we are not overriding existing data but creating new. [See some examples](#examples)
117110
1. Your new code should be able to write to your old and new resources which ensures that we can roll back to the previous state and prevent possible data gaps.
118111
1. Create a pull request and deploy it to every stage in your application
119112

120-
#### The Second Phase (data transformation)
113+
#### 2nd Phase (data transformation)
121114

122115
1. For the first time use `sls dynamodt init` it will generate a folder per table inside the root folder of your service (The name of the folder is the exact name of the table).
123116
A template data transformation file (v1.js) will be created in each table folder. \
@@ -138,13 +131,13 @@ Implement these functions:
138131
Note that the data transformation runs after an sls deploy command it is integrated \
139132
with lifecycle of serverless `after:deploy:deploy` hook.
140133

141-
#### The Third Phase (Use The New Resources/Data):
134+
#### 3rd Phase (Use The New Resources/Data)
142135
1. Adjust your code to work with the new data. \
143136
For example, read from the new index instead of the old one.
144137
1. Create a pull request with the updated lambdas.
145138

146139

147-
#### The Fourth Phase (Cleanup):
140+
#### 4th Phase (Cleanup)
148141
1. Clean the unused data (attributes/indexes/etc).
149142

150143

@@ -154,21 +147,7 @@ First of all, keep in mind that our mission is to prevent downtime while executi
154147
- Your code should be able to work with the old version of the data and keep it updated.
155148
- To be continued...
156149

157-
### Troubleshooting
158-
Credentials error:
159-
#### Required environment variables: #Move it to the tool docs
160-
```bash
161-
# configure aws credentials aws configure or use environment variables
162-
export AWS_ACCESS_KEY_ID=<your-access-key-id>
163-
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
164-
export AWS_DEFAULT_REGION=<your-region>
165-
166-
#[OPTIONAL]
167-
# Required for preparing data for the migration
168-
# The default is migrations-preparation-data
169-
# Make sure you have created the bucket before running the migration
170-
export PREPARATION_DATA_BUCKET=<your-bucket-name>
171-
```
150+
172151

173152
### Data Transformation Script Format (e.g v1_script.js)
174153
```js
@@ -235,25 +214,25 @@ module.exports = {
235214

236215
#### Add a new field to each record
237216
```js
238-
// Adding a "randotNumber" field to each item
217+
// Adding a "randomNumber" field to each item
239218
const { utils } = require('dynamo-data-transform');
240219

241220
const TABLE_NAME = 'UsersExample';
242221

243222
const transformUp = async ({ ddb, isDryRun }) => {
244-
const addRandotNumberField = (item) => {
245-
const updatedItem = { ...item, randotNumber: Math.random() };
223+
const addRandomNumberField = (item) => {
224+
const updatedItem = { ...item, randomNumber: Math.random() };
246225
return updatedItem;
247226
};
248-
return utils.transformItems(ddb, TABLE_NAME, addRandotNumberField, isDryRun);
227+
return utils.transformItems(ddb, TABLE_NAME, addRandomNumberField, isDryRun);
249228
};
250229

251230
const transformDown = async ({ ddb, isDryRun }) => {
252-
const removeRandotNumberField = (item) => {
253-
const { randotNumber, ...oldItem } = item;
231+
const removeRandomNumberField = (item) => {
232+
const { randomNumber, ...oldItem } = item;
254233
return oldItem;
255234
};
256-
return utils.transformItems(ddb, TABLE_NAME, removeRandotNumberField, isDryRun);
235+
return utils.transformItems(ddb, TABLE_NAME, removeRandomNumberField, isDryRun);
257236
};
258237

259238
module.exports = {

examples/serverless-localstack/data-transformations/UsersExample/v2_add_random_number_field.js

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,22 @@
1-
// Adding a "randotNumber" field to each item
1+
// Adding a "randomNumber" field to each item
22
const { utils } = require('dynamo-data-transform');
33

44
const TABLE_NAME = 'UsersExample';
55

66
const transformUp = async ({ ddb, isDryRun }) => {
7-
const addRandotNumberField = (item) => {
8-
const updatedItem = { ...item, randotNumber: Math.random() };
7+
const addRandomNumberField = (item) => {
8+
const updatedItem = { ...item, randomNumber: Math.random() };
99
return updatedItem;
1010
};
11-
return utils.transformItems(ddb, TABLE_NAME, addRandotNumberField, isDryRun);
11+
return utils.transformItems(ddb, TABLE_NAME, addRandomNumberField, isDryRun);
1212
};
1313

1414
const transformDown = async ({ ddb, isDryRun }) => {
15-
const removeRandotNumberField = (item) => {
16-
const { randotNumber, ...oldItem } = item;
15+
const removeRandomNumberField = (item) => {
16+
const { randomNumber, ...oldItem } = item;
1717
return oldItem;
1818
};
19-
return utils.transformItems(ddb, TABLE_NAME, removeRandotNumberField, isDryRun);
19+
return utils.transformItems(ddb, TABLE_NAME, removeRandomNumberField, isDryRun);
2020
};
2121

2222
module.exports = {

0 commit comments

Comments
 (0)