Skip to content

Commit d4d1ce1

Browse files
datakitchen-devopsdatakitchen-devops
authored
Merge pull request #28 from DataKitchen/release/3.1.0
Release/3.1.0
2 parents 0a4b4c7 + bb40884 commit d4d1ce1

23 files changed

Lines changed: 648 additions & 12 deletions

File tree

‎.github/actions/publish_charts/action.yaml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Publish Charts
2+
description: Push charts to GitHub Pages
3+
inputs:
4+
github-token:
5+
description: 'GitHub token for pushing to gh-pages branch'
6+
required: true
7+
runs:
8+
using: 'composite'
9+
steps:
10+
- name: Configure Git
11+
shell: bash
12+
run: |
13+
git config user.name "$GITHUB_ACTOR"
14+
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
15+
16+
- name: Install Helm
17+
uses: azure/setup-helm@v4.2.0
18+
19+
- name: Add Helm Repos for Dependencies
20+
shell: bash
21+
run: |
22+
helm repo add bitnami https://charts.bitnami.com/bitnami
23+
24+
- name: Run chart-releaser
25+
uses: helm/chart-releaser-action@v1.6.0
26+
with:
27+
charts_dir: deploy/charts
28+
skip_existing: 'true'
29+
packages_with_index: 'true'
30+
env:
31+
CR_TOKEN: '${{ inputs.github-token }}'

‎.github/workflows/release.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
name: Release New Version
2+
on:
3+
push:
4+
branches:
5+
- main
6+
workflow_dispatch:
7+
branches:
8+
- main
9+
jobs:
10+
publish:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Checkout Latest Changes
14+
uses: actions/checkout@v4
15+
with:
16+
fetch-depth: 0
17+
18+
- name: Package and Publish Charts
19+
uses: ./.github/actions/publish_charts
20+
with:
21+
github-token: '${{ secrets.GITHUB_TOKEN }}'

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ share/python-wheels/
3030
.installed.cfg
3131
*.egg
3232
MANIFEST
33+
deploy/charts/*/charts
3334

3435
# Installer logs
3536
pip-log.txt

‎deploy/charts/README.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Overview
2+
3+
These are the Helm charts to install DataOps TestGen on a Kubernetes
4+
environment. This README includes instructions for installing TestGen on a
5+
minikube instance, including the application database engine, added by the
6+
`testgen-services` charts. The application itself is installed by the
7+
`testgen-app` charts. When installing on cloud, it's likely that the
8+
application database will be provisioned by a cloud provider, and therefore
9+
installing the `testgen-services` charts is not necessary.
10+
11+
# Preparing the Environment
12+
13+
Here are the instructions to create a local minikube cluster where DataOps
14+
TestGen and its services will be installed. When deploying to a cloud /
15+
production environment, this step should be adjusted accordingly.
16+
17+
Assuming that minikube is installed, the following command will create a
18+
cluster named dk-testgen and configure the local kubectl tool to issue commands
19+
against the cluster. When running helm commands against an existing cluster,
20+
it's important to make sure that the kubectl tool is pointing to the correct
21+
profile.
22+
23+
```shell
24+
minikube start -p dk-testgen --namespace datakitchen
25+
```
26+
27+
# Configuration
28+
29+
Whenever a helm command is used to generate manifests from templates
30+
(`install`, `upgrade`), it will rely on pre-configured values to do so. Helm
31+
provides a variety of ways to define these values. You can find the complete
32+
configuration set that is also used as a default in each charts’ folder, in the
33+
values.yaml file.
34+
35+
No additional configuration is recommended for the `testgen-services` charts.
36+
37+
The `testgen-app` charts require some configuration to be fine tuned. The best
38+
way to do that is to save it in a values file, so that the same configuration
39+
set can be easily used on the first install and future upgrades.
40+
41+
The following configuration is recommended for experimental installations, but
42+
you're free to adjust it for your needs. The next installation steps assumes
43+
that a file named tg-values.yaml exists with this configuration.
44+
45+
```yaml
46+
testgen:
47+
48+
# Password that will be assigned to the 'admin' user during the database preparation
49+
uiPassword: "admin"
50+
51+
# Whether to trust the target database certificates.
52+
trustTargetDatabaseCertificate: true
53+
54+
# Whether to run the SSL certificate verifications when connecting to DataOps Observability
55+
observabilityVerifySsl: false
56+
57+
image:
58+
59+
# DataOps TestGen version to be installed / upgraded
60+
tag: v3.0
61+
```
62+
63+
# Installing
64+
65+
The following command will install the required services for DataOps TestGen,
66+
which currently is the Postgres database. This step is not needed when the
67+
database is provisioned in a cloud environment.
68+
69+
```shell
70+
helm install --create-namespace --wait testgen-services deploy/charts/testgen-services/
71+
```
72+
73+
The following command will install the application. As part of the process, a
74+
one-time database configuration will be automatically performed. Note that the
75+
custom configuration values are being used.
76+
77+
```shell
78+
helm install --wait -f tg-values.yaml testgen-app deploy/charts/testgen-app/
79+
```
80+
81+
At this point you should have a fully functional instance of DataOps TestGen
82+
installed. If you're looking to secure the UI password, you can delete it from
83+
the values file.
84+
85+
# Accessing the Application
86+
87+
In order to use your TestGen instance, you have to use your web browser to
88+
login to it. Your username will be admin and the password the one you
89+
configured earlier.
90+
91+
It may be needed that you forward the application HTTP port in order to be able
92+
to access it. This is especially necessary if you are using the docker driver
93+
for minikube installed into a mac OS. The following command forwards the TesGen
94+
“http” port to the host's 8501 port.
95+
96+
```shell
97+
kubectl port-forward svc/testgen-app 8501:http
98+
```
99+
100+
# Upgrading
101+
102+
When you already have a running instance of TestGen and want to upgrade it to a
103+
newer version, you can do so without losing any data by running the
104+
`upgrade-system-version` TestGen command from the target version's image. The
105+
helm charts will do this automatically when you upgrade. You should edit your
106+
values file to update the image tag value to the desired version, and then run
107+
the following command.
108+
109+
```shell
110+
helm upgrade --wait -f tg-values.yaml testgen-app deploy/charts/testgen-app/
111+
```
112+
113+
# Uninstalling
114+
115+
The following commands will uninstall almost everything that was created
116+
through the helm templates. It usually preserves data, which is helpful to
117+
avoid unplanned data loss, but will likely break a future install attempt.
118+
119+
```shell
120+
helm uninstall testgen-app
121+
```
122+
123+
```shell
124+
helm uninstall testgen-services
125+
```
126+
127+
If you're uninstalling to re-install from scratch, and the data previously
128+
generated can be purged, issue the following command to delete the database
129+
volume so that the next installation will re-create and re-populate it
130+
successfully.
131+
132+
```shell
133+
kubectl delete pvc data-testgen-services-postgresql-0
134+
```

‎deploy/charts/testgen-app/Chart.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
apiVersion: v2
2+
name: testgen
3+
description: A Helm chart for Kubernetes
4+
5+
# A chart can be either an 'application' or a 'library' chart.
6+
#
7+
# Application charts are a collection of templates that can be packaged into versioned archives
8+
# to be deployed.
9+
#
10+
# Library charts provide useful utilities or functions for the chart developer. They're included as
11+
# a dependency of application charts to inject those utilities and functions into the rendering
12+
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
13+
type: application
14+
15+
# This is the chart version. This version number should be incremented each time you make changes
16+
# to the chart and its templates, including the app version.
17+
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18+
version: 0.1.0
19+
20+
# This is the version number of the application being deployed. This version number should be
21+
# incremented each time you make changes to the application. Versions are not expected to
22+
# follow Semantic Versioning. They should reflect the version the application is using.
23+
# It is recommended to use it with quotes.
24+
appVersion: "v3"

‎deploy/charts/testgen-app/templates/_environment.yaml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{{- define "testgen.environment" -}}
2+
- name: TG_DECRYPT_SALT
3+
valueFrom:
4+
secretKeyRef:
5+
name: {{ .Values.testgen.databaseDecryptSecret.name | quote }}
6+
key: "decrypt-salt"
7+
- name: TG_DECRYPT_PASSWORD
8+
valueFrom:
9+
secretKeyRef:
10+
name: {{ .Values.testgen.databaseDecryptSecret.name | quote }}
11+
key: "decrypt-password"
12+
- name: TG_METADATA_DB_HOST
13+
value: {{ .Values.testgen.databaseHost | quote }}
14+
- name: TG_METADATA_DB_NAME
15+
value: {{ .Values.testgen.databaseName | quote }}
16+
- name: TG_METADATA_DB_SCHEMA
17+
value: {{ .Values.testgen.databaseSchema | quote }}
18+
- name: TG_METADATA_DB_USER
19+
value: {{ .Values.testgen.databaseUser | quote }}
20+
- name: TG_METADATA_DB_PASSWORD
21+
valueFrom:
22+
secretKeyRef:
23+
name: {{ .Values.testgen.databasePasswordSecret.name | quote }}
24+
key: {{ .Values.testgen.databasePasswordSecret.key | quote }}
25+
- name: TG_TARGET_DB_TRUST_SERVER_CERTIFICATE
26+
value: {{ .Values.testgen.trustTargetDatabaseCertificate | ternary "yes" "no" | quote }}
27+
- name: TG_EXPORT_TO_OBSERVABILITY_VERIFY_SSL
28+
value: {{ .Values.testgen.observabilityVerifySsl | ternary "yes" "no" | quote }}
29+
- name: TG_DOCKER_RELEASE_CHECK_ENABLED
30+
value: {{ .Values.testgen.releaseCheck | ternary "yes" "no" | quote }}
31+
{{- end -}}
32+
33+
{{- define "testgen.hookEnvironment" -}}
34+
{{- if .Values.testgen.uiPassword -}}
35+
- name: TESTGEN_USERNAME
36+
value: {{ .Values.testgen.uiUser | quote }}
37+
- name: TESTGEN_PASSWORD
38+
value: {{ .Values.testgen.uiPassword | quote }}
39+
{{- end -}}
40+
{{- end -}}

‎deploy/charts/testgen-app/templates/_helpers.tpl‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{{/*
2+
Expand the name of the chart.
3+
*/}}
4+
{{- define "testgen.name" -}}
5+
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
6+
{{- end }}
7+
8+
{{/*
9+
Create a default fully qualified app name.
10+
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
11+
If release name contains chart name it will be used as a full name.
12+
*/}}
13+
{{- define "testgen.fullname" -}}
14+
{{- if .Values.fullnameOverride }}
15+
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
16+
{{- else }}
17+
{{- $name := default .Chart.Name .Values.nameOverride }}
18+
{{- if contains $name .Release.Name }}
19+
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
20+
{{- else }}
21+
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
22+
{{- end }}
23+
{{- end }}
24+
{{- end }}
25+
26+
{{/*
27+
Create chart name and version as used by the chart label.
28+
*/}}
29+
{{- define "testgen.chart" -}}
30+
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
31+
{{- end }}
32+
33+
{{/*
34+
Common labels
35+
*/}}
36+
{{- define "testgen.labels" -}}
37+
helm.sh/chart: {{ include "testgen.chart" . }}
38+
{{ include "testgen.selectorLabels" . }}
39+
{{- if .Chart.AppVersion }}
40+
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
41+
{{- end }}
42+
app.kubernetes.io/managed-by: {{ .Release.Service }}
43+
{{- end }}
44+
45+
{{/*
46+
Selector labels
47+
*/}}
48+
{{- define "testgen.selectorLabels" -}}
49+
app.kubernetes.io/name: {{ include "testgen.name" . }}
50+
app.kubernetes.io/instance: {{ .Release.Name }}
51+
{{- end }}
52+
53+
{{/*
54+
Create the name of the service account to use
55+
*/}}
56+
{{- define "testgen.serviceAccountName" -}}
57+
{{- if .Values.serviceAccount.create }}
58+
{{- default (include "testgen.fullname" .) .Values.serviceAccount.name }}
59+
{{- else }}
60+
{{- default "default" .Values.serviceAccount.name }}
61+
{{- end }}
62+
{{- end }}

‎deploy/charts/testgen-app/templates/deployment.yaml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: {{ include "testgen.fullname" . }}
5+
labels:
6+
{{- include "testgen.labels" . | nindent 4 }}
7+
spec:
8+
replicas: {{ .Values.replicaCount }}
9+
selector:
10+
matchLabels:
11+
{{- include "testgen.selectorLabels" . | nindent 6 }}
12+
template:
13+
metadata:
14+
annotations:
15+
# Used to force Helm to trigger a re-deploy despite this manifest haven't changed
16+
timestamp: {{ now | quote }}
17+
{{- with .Values.podAnnotations }}
18+
{{- toYaml . | nindent 8 }}
19+
{{- end }}
20+
labels:
21+
{{- include "testgen.labels" . | nindent 8 }}
22+
{{- with .Values.podLabels }}
23+
{{- toYaml . | nindent 8 }}
24+
{{- end }}
25+
spec:
26+
{{- with .Values.imagePullSecrets }}
27+
imagePullSecrets:
28+
{{- toYaml . | nindent 8 }}
29+
{{- end }}
30+
serviceAccountName: {{ include "testgen.serviceAccountName" . }}
31+
securityContext:
32+
{{- toYaml .Values.podSecurityContext | nindent 8 }}
33+
containers:
34+
- name: {{ .Chart.Name }}
35+
securityContext:
36+
{{- toYaml .Values.securityContext | nindent 12 }}
37+
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
38+
imagePullPolicy: {{ .Values.image.pullPolicy }}
39+
ports:
40+
- name: http
41+
containerPort: {{ .Values.service.port }}
42+
protocol: TCP
43+
resources:
44+
{{- toYaml .Values.resources | nindent 12 }}
45+
env:
46+
{{- include "testgen.environment" . | nindent 12 }}
47+
{{- with .Values.extraVolumeMounts }}
48+
volumeMounts:
49+
{{ toYaml . | nindent 12 }}
50+
{{- end }}
51+
{{- with .Values.extraVolumes }}
52+
volumes:
53+
{{ toYaml . | nindent 8 }}
54+
{{- end }}
55+
{{- with .Values.nodeSelector }}
56+
nodeSelector:
57+
{{- toYaml . | nindent 8 }}
58+
{{- end }}
59+
{{- with .Values.affinity }}
60+
affinity:
61+
{{- toYaml . | nindent 8 }}
62+
{{- end }}
63+
{{- with .Values.tolerations }}
64+
tolerations:
65+
{{- toYaml . | nindent 8 }}
66+
{{- end }}

0 commit comments

Comments
 (0)