Skip to content

Commit deecdfe

Browse files
author
Federico Rossi
committed
Merge branch '1-riorganizzazione-repo' into 'master'
Resolve "riorganizzazione repo" Closes #1 See merge request openk9/connectors/database-datasource!1
2 parents 76a7143 + 0c01928 commit deecdfe

32 files changed

Lines changed: 427 additions & 812 deletions

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,3 +155,5 @@ target
155155

156156
public
157157

158+
venv
159+
.venv

database-parser/python/app/data/__init__.py renamed to chart/README.md

File renamed without changes.

‎connector/.dockerignore‎

Whitespace-only changes.

‎connector/Dockerfile‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
FROM python:3.12
2+
3+
COPY ./connector/requirements.txt /requirements.txt
4+
5+
# Install utils
6+
RUN apt-get update \
7+
&& apt-get install -y wget \
8+
&& apt-get install -y unzip \
9+
&& apt-get install -y dos2unix \
10+
&& rm -rf /var/lib/apt/lists/*
11+
12+
RUN pip3 install -r requirements.txt
13+
14+
ADD ./connector/docker-entrypoint.sh /etc/docker-entrypoint.sh
15+
16+
COPY ./connector/app /app
17+
18+
WORKDIR /app
19+
20+
RUN dos2unix /etc/docker-entrypoint.sh
21+
22+
RUN chmod a+x /etc/docker-entrypoint.sh
23+
24+
ENTRYPOINT ["/bin/bash"]
25+
26+
CMD ["/etc/docker-entrypoint.sh"]

‎connector/README.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Database Connector
2+
3+
Database connector is a service for extracting data from specific databases.\
4+
Run container from built image and configure appropriate plugin to call it.
5+
6+
The container takes via environment variable INGESTION_URL, which must match the url of the Ingestion Api.
7+
8+
## Database Api
9+
10+
This Rest service exposes one endpoint:
11+
12+
13+
### Execute Database enpoint
14+
15+
Call this endpoint to execute a crawler that extract tables from Database
16+
17+
This endpoint takes different arguments in JSON raw body:
18+
19+
- **dialect**: Database name such as 'mysql', 'oracle', 'postgresql', etc. (required)
20+
- **driver**: Database driver (e.g., psycopg2, pymysql). (required)
21+
- **username**: Database username. (required)
22+
- **password**: Database password. (required)
23+
- **host**: Database server address (IP or domain). (required)
24+
- **port**: Port number for database connection. (required)
25+
- **db**: Name of the database to connect to. (required)
26+
- **table**: Name of the table to extract (required)
27+
- **columns**: Name of the columns to extract (optional, if not specified extract all columns)
28+
- **where**: Where condition used for extraction (optional, if not specified extract without conditions)
29+
- **datasourceId**: id of datasource
30+
- **tenantId**: id of tenant
31+
- **scheduleId**: id of schedulation
32+
- **timestamp**: timestamp to check data to be extracted
33+
34+
Follows an example of Curl call:
35+
36+
```
37+
curl --location --request POST 'http://localhost:5000/getData' \
38+
--header 'Content-Type: application/json' \
39+
--data-raw '{
40+
"dialect": "mysql",
41+
"driver": pymysql,
42+
"username": "admin",
43+
"password": "password",
44+
"host": "localhost",
45+
"port": "8080",
46+
"db": "mydb",
47+
"table": "test_table",
48+
"columns": [],
49+
"where": "",
50+
"datasourceId": 1,
51+
"tenantId": "1",
52+
"scheduleId": "1",
53+
"timestamp": 0
54+
}'
55+
```
56+
57+
### Health check endpoint
58+
59+
Call this endpoint to perform health check for service.
60+
61+
Follows an example of Curl call:
62+
63+
```
64+
curl --location --request POST 'http://localhost:5000/health'
65+
```
66+
67+
### Get sample endpoint
68+
69+
Call this endpoint to get a sample of result.
70+
71+
Follows an example of Curl call:
72+
73+
```
74+
curl --location --request POST 'http://localhost:500/sample'
75+
```
76+
77+
# Quickstart
78+
79+
## How to run
80+
81+
## Docker
82+
83+
### Using Dockerfile
84+
85+
Using the command line go in the Database-datasource parent folder\
86+
From this folder:
87+
```
88+
cd ..
89+
```
90+
91+
Build the Docker file:
92+
```
93+
docker build -t database-parser -f .\connector\Dockerfile .
94+
```
95+
96+
**Command parameters:
97+
- **-t**: Set built image name
98+
- **-f**: Specify the path to the Dockerfile**
99+
100+
Run the built Docker image:
101+
```
102+
docker run -p 5000:5000 --name database-parser-app Database-parser
103+
```
104+
105+
Command parameters:
106+
- **-p**: Exposed port to make api calls
107+
- **-name**: Set docker container name
108+
109+
## Kubernetes
110+
111+
#### TO-DO: Add guide
112+
113+
# Docs and resources
114+
115+
#### TO-DO: Add docs
116+
117+
# Migration Guides
118+
119+
#### TO-DO: Add wiki links
120+
121+
# Contributing
122+
123+
#### TO-DO: Add Contributing
124+
125+
# Authors and acknowledgment
126+
127+
#### TO-DO: Add Authors and acknowledgment
128+
129+
# License
130+
131+
#### TO-DO: Add License

‎connector/app/data/form.json‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
{
2+
"fields": [
3+
{
4+
"label": "Dialect",
5+
"field": "dialect",
6+
"type": "text",
7+
"size": 4,
8+
"required": true,
9+
"values": [],
10+
"info": "Database name such as 'mysql', 'oracle', 'postgresql', etc.",
11+
"placeholder": "mysql",
12+
"validator": {
13+
"min": 0,
14+
"max": 100,
15+
"regex": "/[[:alnum:]]+/"
16+
}
17+
},
18+
{
19+
"label": "Driver",
20+
"field": "driver",
21+
"type": "text",
22+
"size": 4,
23+
"required": true,
24+
"values": [],
25+
"info": "Database driver (e.g., psycopg2, pymysql).",
26+
"placeholder": "pymysql",
27+
"validator": {
28+
"min": 0,
29+
"max": 100,
30+
"regex": "/[[:alnum:]]+/"
31+
}
32+
},
33+
{
34+
"label": "Username",
35+
"field": "username",
36+
"type": "text",
37+
"size": 4,
38+
"required": true,
39+
"values": [],
40+
"info": "Database username.",
41+
"placeholder": "admin",
42+
"validator": {
43+
"min": 0,
44+
"max": 100,
45+
"regex": "/[[:alnum:]]+/"
46+
}
47+
},
48+
{
49+
"label": "Password",
50+
"name": "password",
51+
"type": "password",
52+
"size": 4,
53+
"required": true,
54+
"values": [],
55+
"info": "Database password.",
56+
"placeholder": "password",
57+
"validator": {
58+
"min": 0,
59+
"max": 100,
60+
"regex": "/[[:alnum:]]+/"
61+
}
62+
},
63+
{
64+
"label": "Host",
65+
"field": "host",
66+
"type": "text",
67+
"size": 4,
68+
"required": true,
69+
"values": [],
70+
"info": "Database server address (IP or domain).",
71+
"placeholder": "localhost",
72+
"validator": {
73+
"min": 0,
74+
"max": 100,
75+
"regex": "/[[:alnum:]]+/"
76+
}
77+
},
78+
{
79+
"label": "Port",
80+
"field": "port",
81+
"type": "text",
82+
"size": 4,
83+
"required": true,
84+
"values": [],
85+
"info": "Port number for database connection.",
86+
"placeholder": "8080",
87+
"validator": {
88+
"min": 0,
89+
"max": 100,
90+
"regex": "/[[:alnum:]]+/"
91+
}
92+
},
93+
{
94+
"label": "Database",
95+
"field": "db",
96+
"type": "text",
97+
"size": 4,
98+
"required": true,
99+
"values": [],
100+
"info": "Name of the database to connect to.",
101+
"placeholder": "mydb",
102+
"validator": {
103+
"min": 0,
104+
"max": 100,
105+
"regex": "/[[:alnum:]]+/"
106+
}
107+
},
108+
{
109+
"label": "Table",
110+
"field": "table",
111+
"type": "text",
112+
"size": 4,
113+
"required": true,
114+
"values": [],
115+
"info": "Name of the table to extract",
116+
"placeholder": "TableName",
117+
"validator": {
118+
"min": 0,
119+
"max": 100,
120+
"regex": "/[[:alnum:]]+/"
121+
}
122+
},
123+
{
124+
"label": "Columns",
125+
"field": "columns",
126+
"type": "list",
127+
"size": 4,
128+
"required": false,
129+
"values": [],
130+
"info": "Name of the columns to extract",
131+
"validator": {
132+
"min": 0,
133+
"max": 100,
134+
"regex": "/[[:alnum:]]+/"
135+
}
136+
},
137+
{
138+
"label": "Where condition",
139+
"field": "where",
140+
"type": "text",
141+
"size": 4,
142+
"required": false,
143+
"values": [],
144+
"info": "Where condition used for extraction",
145+
"validator": {
146+
"min": 0,
147+
"max": 100,
148+
"regex": "/[[:alnum:]]+/"
149+
}
150+
},
151+
{
152+
"label": "Timestamp",
153+
"name": "timestamp",
154+
"type": "number",
155+
"size": 4,
156+
"required": true,
157+
"values": [
158+
{
159+
"value": 0,
160+
"isDefault": true
161+
}
162+
],
163+
"placeholder": "",
164+
"info": "",
165+
"validator": {
166+
"min": 0,
167+
"max": 100,
168+
"regex": "/[[:alnum:]]+/"
169+
}
170+
},
171+
{
172+
"label": "Datasource Id",
173+
"name": "datasourceId",
174+
"type": "number",
175+
"size": 4,
176+
"required": true,
177+
"values": [
178+
{
179+
"value": 1,
180+
"isDefault": true
181+
}
182+
],
183+
"placeholder": "",
184+
"info": "",
185+
"validator": {
186+
"min": 0,
187+
"max": 100,
188+
"regex": "/[[:alnum:]]+/"
189+
}
190+
},
191+
{
192+
"label": "Schedule Id",
193+
"name": "scheduleId",
194+
"type": "text",
195+
"size": 4,
196+
"required": true,
197+
"values": [
198+
{
199+
"value": "1",
200+
"isDefault": true
201+
}
202+
],
203+
"placeholder": "",
204+
"info": "",
205+
"validator": {
206+
"min": 0,
207+
"max": 100,
208+
"regex": "/[[:alnum:]]+/"
209+
}
210+
},
211+
{
212+
"label": "Tenant Id",
213+
"name": "tenantId",
214+
"type": "text",
215+
"size": 4,
216+
"required": true,
217+
"values": [
218+
{
219+
"value": "1",
220+
"isDefault": true
221+
}
222+
],
223+
"placeholder": "",
224+
"info": "",
225+
"validator": {
226+
"min": 0,
227+
"max": 100,
228+
"regex": "/[[:alnum:]]+/"
229+
}
230+
}
231+
]
232+
}

0 commit comments

Comments
 (0)