Skip to content

Commit 20733af

Browse files
authored
Merge pull request #1 from commitdev/add-circleci-pipeline
Added circleci pipeline from commit0-ci-pipeline project, with some t…
2 parents 02188d7 + 5385f69 commit 20733af

5 files changed

Lines changed: 384 additions & 0 deletions

File tree

.circleci/README.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# CircleCI Pipeline Template
2+
3+
## Configuration
4+
5+
### Requirements
6+
7+
Requires you to configure the below [CircleCI Environment Variables](https://circleci.com/docs/2.0/env-vars/):
8+
9+
- AWS_DEFAULT_REGION # Region of your cluster
10+
- AWS_ECR_ACCOUNT_URL # {awsAccountNum}.dkr.ecr.{region}.amazonaws.com
11+
- AWS_ECR_REPO_NAME # The ECR repository name to write images to
12+
- PRODUCTION_EKS_CLUSTER_NAME # The name of the production EKS cluster to deploy into
13+
- STAGING_EKS_CLUSTER_NAME # The name of the staging EKS cluster to deploy into
14+
15+
16+
## Deployment Process
17+
18+
### Branch Deploys
19+
20+
1. [Checkout](#checkout)
21+
2. [Unit Tests](#unit-tests)
22+
23+
### Master Deploys
24+
25+
1. [Checkout](#checkout)
26+
2. [Unit Tests](unit-tests)
27+
3. [Build and Push Image](#build-and-push-image)
28+
4. [Deploy Staging](#deploy-staging)
29+
5. [Integration Tests](#integration-test)
30+
6. [Deploy Production](#deploy-production)
31+
32+
## Checkout
33+
34+
We checkout code in a separate step and then save to a shared workspace throughout the rest of the build. This is done to avoid needing to run the checkout in multiple steps throughout the build.
35+
36+
## Unit Tests
37+
38+
We run the tests inside a Go container. This runs parallel to the build process to save time. We run unit tests on branches before merging to master so these should never fail on the master branch. Results are piped into go-junit-reporter to allow CircleCI to parse them for [Insights](https://circleci.com/build-insights/gh/Vin65/shipping-service/master).
39+
40+
## Build and Push Image
41+
42+
Uses the ECR orb (See: [Orbs](#orbs)) to build the Dockerfile and push the image up to ECR.
43+
44+
## Deploy Staging
45+
46+
Still finishing up with this, currently it just runs the command `make deploy-staging`
47+
48+
## Integration Tests
49+
50+
Set of tests that need to run against a working version of the application. These are run after deploying to a staging or testing server.
51+
52+
## Deploy Production
53+
54+
This does the same EKS deployment, just waits for other builds to finish deploying first.
55+
56+
## Orbs
57+
58+
### AWS
59+
60+
- [ECR Orb](https://circleci.com/orbs/registry/orb/circleci/aws-ecr)
61+
- [EKS Orb](https://circleci.com/orbs/registry/orb/circleci/aws-eks)
62+
- [CLI Orb](https://circleci.com/orbs/registry/orb/circleci/aws-cli)
63+
64+
We use multiple AWS orbs to simplify the deployment process. Firstly to build the image and push to the repo, we use the `aws-ecr` orb. This just combines the commands needed to build a docker image, tag it, and push to ECR. The tags are dealt with by the `version-tag` orb (see below).
65+
66+
### Slack
67+
68+
- [Slack Orb](https://circleci.com/orbs/registry/orb/circleci/slack)
69+
70+
You'll need to set up a slack webhook to post messages to a channel in Slack. You can do it in the [Incoming Webhooks](https://winedirectteam.slack.com/apps/A0F7XDUAZ-incoming-webhooks?next_id=0) page. This is then added as an environment variable on your project with the name `SLACK_WEBHOOK` as per the [Slack Orb Documentation](https://circleci.com/orbs/registry/orb/circleci/slack).
71+
72+
### Commit Version Tag
73+
74+
- [Version Tag Orb](https://circleci.com/orbs/registry/orb/commitdev/version-tag)
75+
76+
This is a custom orb used by (Commit)[https://commit.dev] to create a tag for docker images that is easier to work with. Instead of using the full Git SHA
77+
as the docker image tag, we use the build number from CircleCI and a short version of the Git SHA (first seven characters only). You'll find the tag for each build in the "Create Version Tag" step inside the `build_and_push` job.
78+
79+
E.g. `Created version tag: VERSION_TAG=164-27a3d39`
80+
81+
This is done to increase the readability, make the numbers sortable and still guarantee uniqueness with a high degree of certainty.
82+
83+
### Queue
84+
85+
- [EddieWebb Queue Orb](https://circleci.com/orbs/registry/orb/eddiewebb/queue)
86+
87+
This orb is used to prevent multiple deployments going out at the same time. By using the CIRCLECI_API_KEY to access the current running builds, it's able to check that the current build is the oldest in the queue, and therefore is able to run first. Other builds in the queue will be put into a wait/retry loop until all builds before them have finished.
88+
89+
This is run before the deploy production step to prevent two builds deploying to production at the same time.
90+
91+
```yaml
92+
- queue/block_workflow:
93+
time: '30' # hold for 30 mins then abort
94+
requires:
95+
- wait_for_approval
96+
```

.circleci/config.yml

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
---
2+
version: 2.1
3+
orbs:
4+
aws-ecr: circleci/aws-ecr@6.5.0
5+
aws-eks: circleci/aws-eks@0.2.3
6+
aws-s3: circleci/aws-s3@1.0.11
7+
aws-cli: circleci/aws-cli@0.1.18
8+
queue: eddiewebb/queue@1.3.0
9+
slack: circleci/slack@3.4.2
10+
version-tag: commitdev/version-tag@0.0.3
11+
12+
variables:
13+
- &workspace /home/circleci/project
14+
15+
- &build-image cimg/go:1.13
16+
17+
18+
aliases:
19+
# Shallow Clone - this allows us to cut the 2 minute repo clone down to about 10 seconds for repos with 50,000 commits+
20+
- &checkout-shallow
21+
name: Checkout (Shallow)
22+
command: |
23+
#!/bin/sh
24+
set -e
25+
26+
# Workaround old docker images with incorrect $HOME
27+
# check https://github.com/docker/docker/issues/2968 for details
28+
if [ "${HOME}" = "/" ]
29+
then
30+
export HOME=$(getent passwd $(id -un) | cut -d: -f6)
31+
fi
32+
33+
mkdir -p ~/.ssh
34+
35+
echo 'github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
36+
bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==' >> ~/.ssh/known_hosts
37+
38+
(umask 077; touch ~/.ssh/id_rsa)
39+
chmod 0600 ~/.ssh/id_rsa
40+
(cat \<<EOF > ~/.ssh/id_rsa
41+
$CHECKOUT_KEY
42+
EOF
43+
)
44+
45+
# use git+ssh instead of https
46+
git config --global url."ssh://git@github.com".insteadOf "https://github.com" || true
47+
48+
if [ -e /home/circleci/project/.git ]
49+
then
50+
cd /home/circleci/project
51+
git remote set-url origin "$CIRCLE_REPOSITORY_URL" || true
52+
else
53+
mkdir -p /home/circleci/project
54+
cd /home/circleci/project
55+
git clone --depth=1 "$CIRCLE_REPOSITORY_URL" .
56+
fi
57+
58+
if [ -n "$CIRCLE_TAG" ]
59+
then
60+
git fetch --depth=10 --force origin "refs/tags/${CIRCLE_TAG}"
61+
elif [[ "$CIRCLE_BRANCH" =~ ^pull\/* ]]
62+
then
63+
# For PR from Fork
64+
git fetch --depth=10 --force origin "$CIRCLE_BRANCH/head:remotes/origin/$CIRCLE_BRANCH"
65+
else
66+
git fetch --depth=10 --force origin "$CIRCLE_BRANCH:remotes/origin/$CIRCLE_BRANCH"
67+
fi
68+
69+
if [ -n "$CIRCLE_TAG" ]
70+
then
71+
git reset --hard "$CIRCLE_SHA1"
72+
git checkout -q "$CIRCLE_TAG"
73+
elif [ -n "$CIRCLE_BRANCH" ]
74+
then
75+
git reset --hard "$CIRCLE_SHA1"
76+
git checkout -q -B "$CIRCLE_BRANCH"
77+
fi
78+
79+
git reset --hard "$CIRCLE_SHA1"
80+
pwd
81+
82+
- &install-binaries
83+
name: Install Binaries
84+
command: |
85+
KUSTOMIZE_VERSION=3.5.4
86+
IAM_AUTH_VERSION=0.5.0
87+
curl -L -o ./kustomize.tar.gz "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv${KUSTOMIZE_VERSION}/kustomize_v${KUSTOMIZE_VERSION}_linux_amd64.tar.gz"
88+
sudo tar xvzf ./kustomize.tar.gz -C /usr/local/bin/
89+
sudo chmod +x /usr/local/bin/kustomize
90+
kustomize version
91+
curl -L -o ./aws-iam-authenticator "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${IAM_AUTH_VERSION}/aws-iam-authenticator_${IAM_AUTH_VERSION}_linux_amd64"
92+
sudo mv ./aws-iam-authenticator /usr/local/bin/
93+
sudo chmod +x /usr/local/bin/aws-iam-authenticator
94+
95+
96+
jobs:
97+
checkout_code:
98+
docker:
99+
- image: *build-image
100+
steps:
101+
- run: *checkout-shallow
102+
- persist_to_workspace:
103+
root: /home/circleci/project
104+
paths:
105+
- .
106+
107+
unit_test:
108+
docker:
109+
- image: *build-image
110+
working_directory: *workspace
111+
steps: # steps that comprise the `build` job
112+
- attach_workspace:
113+
at: *workspace
114+
- restore_cache: # restores saved cache if no changes are detected since last run
115+
keys:
116+
- v1-pkg-cache-{{ checksum "go.sum" }}
117+
- v1-pkg-cache-
118+
- run:
119+
name: Run unit tests
120+
command: |
121+
go get -u github.com/jstemmer/go-junit-report
122+
mkdir -p test-reports
123+
go test -v ./... | go-junit-report > test-reports/junit.xml
124+
125+
- save_cache: # Store cache in the /go/pkg directory
126+
key: v1-pkg-cache-{{ checksum "go.sum" }}
127+
paths:
128+
- "/go/pkg"
129+
130+
- store_test_results:
131+
path: test-reports
132+
133+
- store_artifacts:
134+
path: test-reports
135+
136+
# Requires the SLACK_WEBHOOK
137+
- slack/notify-on-failure
138+
139+
build_and_push:
140+
machine:
141+
docker_layer_caching: false # only for performance plan circleci accounts
142+
steps:
143+
- attach_workspace:
144+
at: *workspace
145+
- run: *checkout-shallow
146+
- version-tag/create
147+
- aws-ecr/build-and-push-image:
148+
repo: ${AWS_ECR_REPO_NAME}
149+
tag: $VERSION_TAG,latest
150+
151+
deploy:
152+
executor: aws-eks/python3
153+
parameters:
154+
namespace:
155+
type: string
156+
default: ''
157+
description: |
158+
The kubernetes namespace that should be used.
159+
repo:
160+
type: string
161+
default: ''
162+
description: |
163+
The name of the ECR repo to deploy an image from.
164+
config-environment:
165+
type: string
166+
default: ''
167+
description: |
168+
The environment kustomize should overlay to generate the kubernetes config. Options are the directories in kubernetes/overlays/
169+
tag:
170+
type: string
171+
default: $VERSION_TAG
172+
description: |
173+
The tag that should be deployed.
174+
region:
175+
type: string
176+
default: ''
177+
description: |
178+
The region to use for AWS operations.
179+
cluster-name:
180+
description: |
181+
The name of the EKS cluster.
182+
type: string
183+
cluster-authentication-role-arn:
184+
default: ''
185+
description: |
186+
To assume a role for cluster authentication, specify an IAM role ARN with
187+
this option. For example, if you created a cluster while assuming an IAM
188+
role, then you must also assume that role to connect to the cluster the
189+
first time.
190+
type: string
191+
steps:
192+
- run: *checkout-shallow
193+
- version-tag/get
194+
- run: *install-binaries
195+
- aws-cli/install
196+
- aws-cli/setup
197+
- aws-eks/update-kubeconfig-with-authenticator:
198+
cluster-name: << parameters.cluster-name >>
199+
cluster-authentication-role-arn: << parameters.cluster-authentication-role-arn >>
200+
aws-region: << parameters.region >>
201+
install-kubectl: true
202+
- queue/until_front_of_line:
203+
time: '30'
204+
- run:
205+
name: Deploy
206+
command: |
207+
cd kubernetes/overlays/<< parameters.config-environment >>
208+
IMAGE=${AWS_ECR_ACCOUNT_URL}/<< parameters.repo >>
209+
kustomize edit set image fake-image=${IMAGE}:${VERSION_TAG}
210+
kustomize build . | kubectl apply -f - -n << parameters.namespace >>
211+
workflows:
212+
version: 2
213+
# The main workflow. Check out the code, build it, push it, deploy to staging, test, deploy to production
214+
build_test_and_deploy:
215+
jobs:
216+
- checkout_code
217+
218+
- unit_test:
219+
requires:
220+
- checkout_code
221+
222+
- build_and_push:
223+
requires:
224+
- unit_test
225+
filters:
226+
branches:
227+
only: # only branches matching the below regex filters will run
228+
- /^master$/
229+
230+
- deploy:
231+
name: deploy_staging
232+
repo: "${AWS_ECR_REPO_NAME}"
233+
cluster-name: "${STAGING_EKS_CLUSTER_NAME}"
234+
config-environment: "staging"
235+
cluster-authentication-role-arn: "${AWS_CLUSTER_AUTH_ROLE_ARN_STAGING}"
236+
region: "${AWS_DEFAULT_REGION}"
237+
namespace: "${CIRCLE_BRANCH}"
238+
tag: "${VERSION_TAG}"
239+
requires:
240+
- build_and_push
241+
242+
- wait_for_approval:
243+
type: approval
244+
requires:
245+
- deploy_staging
246+
247+
- queue/block_workflow:
248+
time: '30' # hold for 30 mins then abort
249+
requires:
250+
- wait_for_approval
251+
252+
- deploy:
253+
name: deploy_production
254+
repo: "${AWS_ECR_REPO_NAME}"
255+
cluster-name: "${PRODUCTION_EKS_CLUSTER_NAME}"
256+
config-environment: "production"
257+
cluster-authentication-role-arn: "${AWS_CLUSTER_AUTH_ROLE_ARN_PRODUCTION}"
258+
region: "${AWS_DEFAULT_REGION}"
259+
namespace: "${CIRCLE_BRANCH}"
260+
tag: "${VERSION_TAG}"
261+
requires:
262+
- queue/block_workflow

commit0.module.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
name: commit0-deployable-backend
2+
description: 'A deployable project including an application, Dockerfile, CircleCI pipeline, and k8s manifests'
3+
author: 'Commit'
4+
icon: ''
5+
thumbnail: ''
6+
7+
template:
8+
strictMode: true
9+
delimiters:
10+
- '<%'
11+
- '%>'
12+
output: '.'

example.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ name: example-app
44
# params are key value pairs passed into templates
55
params:
66

7+
# Application config
8+
79
# production host name
810
productionHost: fix_me
911

main_test.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
package main_test
2+
3+
import (
4+
"testing"
5+
)
6+
7+
func TestExample(t *testing.T) {
8+
got := (1 + 1)
9+
if got != 2 {
10+
t.Errorf("1 + 1 = %d; want 2", got)
11+
}
12+
}

0 commit comments

Comments
 (0)