You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- We design our databases schemas and define how we want to store the data inside. \
11
-
We hope that our design will last forever, \
12
-
however, in the real world, it might not happen. \
13
-
Making changes in our database design is a regular process that happens sometimes.
10
+
Dynamo Data Transform is an easy to use data transformation tool for DynamoDB.
11
+
12
+
It allows performing powerful data transformations using simple Javascript commands, without the risk of breaking your database.
13
+
Available as a [Serverless plugin](#serverless-plugin), [npm package](#standalone-npm-package) and even as an [interactive CLI](#interactive-cli), Dynamo Data Transform saves you time and keeps you safe with features like dry-running a data transformation and even rolling back your last trasnformation if needed.
14
14
15
-
- We need to be able to execute data transformations \
16
-
without interfering with us or our clients \
17
-
and without losing the ability to get back to the previous state of the data.
- Dry run option for each command (by suppling --dry flag, the data will be printed instead of stored).
23
+
- Safe & Secure preparation data
24
+
- Store preparation data in a private s3 bucket. [Prepare data for your data transformation](#usage-and-command-line-options)
20
25
21
-
## Quick Start:
22
-
### Serverless plugin:
26
+
## Quick Start
27
+
### Serverless plugin
23
28
- Install
24
29
```bash
25
30
npm install dynamo-data-transform --save-dev
@@ -34,7 +39,7 @@ plugins:
34
39
sls dynamodt list
35
40
```
36
41
37
-
### Standalone npm package:
42
+
### Standalone npm package
38
43
- Install the tool
39
44
```bash
40
45
npm install -g dynamo-data-transform
@@ -47,37 +52,26 @@ Or with the shortcut
47
52
```bash
48
53
ddt --help
49
54
```
50
-
- Use the interactive cli
55
+
### Interactive CLI
56
+
After installing the npm package, run:
51
57
```bash
52
58
dynamodt -i
53
59
```
54
60
55
61
56
62
57
63
58
-
**Features**
59
64
60
-
- Seemless data transformations management.
61
-
- Support for multiple stages.
62
-
- History of executed data transformations.
63
-
- Dry run option for each command (by suppling --dry flag, the data will be printed instead of stored).
64
-
- Safe & Secure preparation data - \
65
-
Store preparation data in a private s3 bucket. \
66
-
[Prepare data for your data transformation](#usage-and-command-line-options)
67
65
68
66
69
67
70
-
## Documentation
68
+
## Table of contents
71
69
72
-
-[Introduction:](#introduction)
73
-
-[Documentation](#documentation)
74
-
-[Installation](#installation)
70
+
-[Quick Start](#quick-start)
75
71
-[Usage and command-line options](#usage-and-command-line-options)
76
-
-[What it does behind the scenes:](#what-it-does-behind-the-scenes)
77
-
-[The data transformation process:](#the-data-transformation-process)
78
-
-[Steps](#steps)
79
-
-[First Phase:](#first-phase)
80
-
-[The Second Phase (data transformation):](#the-second-phase-data-transformation)
72
+
-[What happens behind the scenes](#what-happens-behind-the-scenes)
73
+
-[The data transformation process](#the-data-transformation-process)
74
+
-[Process Steps](#steps)
81
75
-[Key Concepts](#key-concepts)
82
76
-[Troubleshooting](#troubleshooting)
83
77
-[Examples](#examples)
@@ -103,21 +97,20 @@ Serverless plugin:
103
97
sls dynamotdt <command> --help
104
98
```
105
99
106
-
## What it does behind the scenes
107
-
- When a data transformation is running for the first time a Record in your table is created. \
108
-
This record is for tracking the executed transformations on a specific table.
100
+
## What happens behind the scenes
101
+
- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.
109
102
110
103
111
104
## The safe data transformation process
112
-
The next section describes how the process looks like and the order of every step in the data transformation process.
105
+
The next section describes how the data transformation process looks like, and the order of each step.
113
106
### Steps
114
-
#### First Phase (Add New Resources)
107
+
#### 1st Phase (Add New Resources)
115
108
1. Update the serverless.yml resources (if needed) \
116
109
Reminder: we are not overriding existing data but creating new. [See some examples](#examples)
117
110
1. Your new code should be able to write to your old and new resources which ensures that we can roll back to the previous state and prevent possible data gaps.
118
111
1. Create a pull request and deploy it to every stage in your application
119
112
120
-
#### The Second Phase (data transformation)
113
+
#### 2nd Phase (data transformation)
121
114
122
115
1. For the first time use `sls dynamodt init` it will generate a folder per table inside the root folder of your service (The name of the folder is the exact name of the table).
123
116
A template data transformation file (v1.js) will be created in each table folder. \
@@ -138,13 +131,13 @@ Implement these functions:
138
131
Note that the data transformation runs after an sls deploy command it is integrated \
139
132
with lifecycle of serverless `after:deploy:deploy` hook.
140
133
141
-
#### The Third Phase (Use The New Resources/Data):
134
+
#### 3rd Phase (Use The New Resources/Data)
142
135
1. Adjust your code to work with the new data. \
143
136
For example, read from the new index instead of the old one.
144
137
1. Create a pull request with the updated lambdas.
145
138
146
139
147
-
#### The Fourth Phase (Cleanup):
140
+
#### 4th Phase (Cleanup)
148
141
1. Clean the unused data (attributes/indexes/etc).
149
142
150
143
@@ -154,21 +147,7 @@ First of all, keep in mind that our mission is to prevent downtime while executi
154
147
- Your code should be able to work with the old version of the data and keep it updated.
155
148
- To be continued...
156
149
157
-
### Troubleshooting
158
-
Credentials error:
159
-
#### Required environment variables: #Move it to the tool docs
160
-
```bash
161
-
# configure aws credentials aws configure or use environment variables
0 commit comments