Skip to content

Commit 810bdeb

Browse files
bietkulsiddharthlatest
authored andcommitted
feat: add runtime parity for handling both ElasticSearch 6 and 7 upstreams (#51)
* feat: support es version > 7.0 * feat: add logs * feat: update es version * feat(build): reduce docker image size * fix: PATCH user and permissions endpoint * fix: size issue in logs * feat: add billing * Update main.go * fix: Use tier_validity from accapi * fix: resolve comments * fix: update validity check condition * fix: request logs at index level * fix: billing env & remove email * update accapi * fix: remove subscription id * fix: minor fixes * add logs * fix: minor fixes * fix: update request * fix: remove subscription id * fix: update URL * fix: remove fmt.Println(..) log * fix: minor fixes * fix: minor fixes * fix: update URL * fix: update arc instance endpoint * fix: update URL * feat: allow self signed certs * fix: handle invalid ARC_ID + minor fixes * fix: Rename variable * Refactor: Enhancements + fixes to the billing mechanism (#24) * test: use local ACC_API URL for test * wip: add debug log * wip: add debug log * wip: add debug log * wip: handle case where time validity = 0 * wip: fix time_validity value while a user is in trial * fix: export ArcInstanceDetails and NodeCount - make sure nodes are counted at init time irrespective of a valid subscription_id and report usage call * fix: set type of NodeCount to int * format: remove debug statements * fix: handle case of an invalid ARC_ID * format: improve error message for env not found * fix: Modified query path for Patch Permission test * feat: expose billing variable from util.go (#26) * Add support for custom jwt usename key The usename could be both for users & permissions * Fix indentation * feat: sync to dev * Add support for modifying/deleting role in permission using PATCH request Also add GET handler for getting a permission using role * Fix indentation * feat: allow self signed certs * fix: es7 changes * fixed middleware test of plugin * fix: permission + post user * fix: permission + post user (#29) * RBAC: Add routes to GET/SET public key (#27) * Create CRUD permission handlers for roles * feat: add routes to GET/SET public key * fix: minor fixes * Fix: get public key route if es index is not present (#30) * fix: permission + post user * fix: get public key route if es index is not present * Fixed getCredentialsTest of dao * Fixed minor bug * Fixed putPermissionTest of dao * Fixed jwt token tests * Minor change * Made tests for handlers and completed methods for middleware * Minor changes * feat: support custom headers (#34) * feat: add report usage for hosted cluster (#33) * wip: add flag for hosted billing * feat: add cron for reporting arc cluster usage * fix: remove encryption * Modified tests * fix: allow sources check to pass if all is set * handles IPv6 match case * fix(permission): update referers error message * feat: validate plan * feat: add suggestions category * feat: add limits for missing categories * fix: plan validation for hosted billing * fix: change the plan validation logic * fix: auth header redirection * feat: fetch & set cluster plan * fix: minor fixes * feat: improve validat * fix: minor category fix * fix: minor fixes * fix: suggestions category * fix: minor fixes * fix: update URL * add build flags * fix: minor fixes * feat: apply billing middleware for cluster plans * fix: revert ACC_API url * fix: stop execution if arc instance is not present * fix(billing): hosted arc (#41) * fix: arc_hosted billing * fix: serious billing issue with time_validity * feat: add ingnoreBillingMiddleware flag * fix: revert acc_api * fix: minor fixes * fix: minor changes * fix: docker file (#42) change CMD -> ENTRYPOINT add shared-docker volume path creation * chore: adds default CMD, ignore config files * fix: fallback to highest plan when billing is disabled * fix: minor fixes * mock: proxy getRawLogs() to work with ES v6 and ES v7 * wip: move es client instantiation to a separate file * chore: update to use the new endpoint route * test * fix: use consistent report_usage URI * fix: update report_usage route * fix: time_validity in self-hosted billing * fix: test * fix: blacklist subscription route * fix: blacklist plan route * fix: revert ACC_API * fix(auth): authentication bug with JWT * fix a bug where role name should be queried against .keyword field * fix bug where JWT key from environment variables would override values set in DB * fix a crash where the role key isn't present in JWT claims * format error messages related to username/role authentication for better comprehension * chore: transform POST search request to GET requests (#43) * fix: remove CMD preset * fix: test accapi * fix: remove use of arc_id * fix: revert accapi * fix: set retrier max limit to 8 sec (from 8 ms) * hotfix: add retry logic to ES proxy * feat: use retrable (#46) * feat: add support for source_filtering * feat: apply source filtering for msearch * feat: support es6 in logs plugin * feat: add support for es6 in auth plugin * refactor: parity changes * chore: change the func name * feat: add support for es6 in permissions plugin * feat: add support for es6 in reindexer plugin * feat: add es6 support for users plugin * chore: refactor auth plugin * chore: refactor logs * chore: refactor permissions * fix: warnings * fix: test cases * chore: update to go1.13 * chore: remove test cases + initialize e2e test * fix: GET permissions for es6 * fix: es6 issues in users and permissions * chore: add test cases for permissions * chore: add users plugins test * fix: minor fixes * fix: minor fixes
1 parent fc06cc9 commit 810bdeb

81 files changed

Lines changed: 3020 additions & 3669 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,5 @@ build
66
*.http
77

88
tags
9+
10+
config

Dockerfile

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,55 @@
1-
FROM golang:1.11-alpine as build
1+
FROM golang:1.13-alpine as builder
2+
3+
# Default value
4+
# Run `--build-arg BILLING=true` to enable billing
5+
ARG BILLING=false
6+
ENV BILLING="${BILLING}"
7+
8+
# Run `--build-arg HOSTED_BILLING=true` to enable billing for hosted arc
9+
ARG HOSTED_BILLING=false
10+
ENV HOSTED_BILLING="${HOSTED_BILLING}"
11+
12+
# Run `--build-arg CLUSTER_BILLING=true` to enable billing for clusters
13+
ARG CLUSTER_BILLING=false
14+
ENV CLUSTER_BILLING="${CLUSTER_BILLING}"
15+
16+
# Run `--build-arg IGNORE_BILLING_MIDDLEWARE=true` to disable billing middleware for testing
17+
ARG IGNORE_BILLING_MIDDLEWARE=false
18+
ENV IGNORE_BILLING_MIDDLEWARE="${IGNORE_BILLING_MIDDLEWARE}"
19+
20+
# Run `--build-arg PLAN_REFRESH_INTERVAL=X` to change the default interval of 1 hour, where 'X' is an integer represent the hours unit
21+
ARG PLAN_REFRESH_INTERVAL=1
22+
ENV PLAN_REFRESH_INTERVAL="${PLAN_REFRESH_INTERVAL}"
223

324
# Install tools required for project
425
# Run `docker build --no-cache .` to update dependencies
526
RUN apk add --no-cache build-base git
627
WORKDIR /arc
728

829
# List project dependencies with go.mod and go.sum
9-
COPY go.mod .
10-
COPY go.sum .
30+
COPY go.mod go.sum ./
1131

1232
# Install library dependencies
13-
RUN go mod download
33+
RUN go mod download
1434

1535
# Copy the entire project and build it
1636
# This layer is rebuilt when a file changes in the project directory
1737
COPY . .
1838
RUN make
1939

40+
# Final stage: Create the running container
41+
FROM alpine:3.10.1 AS final
42+
43+
# Get ca certs, for making api calls
44+
RUN apk add --no-cache ca-certificates
45+
46+
47+
# Create env folder
48+
RUN mkdir /arc-data && touch /arc-data/.env && chmod 777 /arc-data/.env
49+
50+
# Import the compiled executable from the first stage.
51+
COPY --from=builder /arc /arc
52+
WORKDIR /arc
53+
2054
EXPOSE 8000
21-
CMD ["build/arc", "--log", "stdout", "--plugins"]
55+
ENTRYPOINT ["build/arc", "--log", "stdout", "--plugins"]

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ PLUGIN_MAIN_LOC_FUNC=plugins/$(1)/main/$(1).$(2)
99
PLUGIN_LOC_FUNC=$(foreach PLUGIN,$(PLUGINS),$(call PLUGIN_MAIN_LOC_FUNC,$(PLUGIN),$(1)))
1010

1111
cmd: plugins
12-
$(GC) -o $(BUILD_DIR)/arc main.go
12+
$(GC) -ldflags "-w -X main.Billing=$(BILLING) -X main.HostedBilling=$(HOSTED_BILLING) -X main.ClusterBilling=$(CLUSTER_BILLING) -X main.PlanRefreshInterval=$(PLAN_REFRESH_INTERVAL) -X main.IgnoreBillingMiddleware=$(IGNORE_BILLING_MIDDLEWARE)" -o $(BUILD_DIR)/arc main.go
1313

1414
plugins: $(call PLUGIN_LOC_FUNC,so)
1515

README.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Arc
22

3-
Arc is a simple, modular API Gateway that sits between a client and an [ElasticSearch](https://elastic.co) cluster. It acts as a reverse proxy, routing requests from clients to services. Arc is extended through plugins, which provide extra functionality and services beyond the ElasticSearch's RESTful API. It can perform various cross-cutting tasks such as basic authentication, logging, rate-limiting, source/referers whitelisting, analytics etc. These functionalities can clearly be extended by adding a plugin encapsulating a desired functionality. It also provides some useful abstractions that helps in managing and controlling the access
3+
Arc is a simple, modular API Gateway that sits between a client and an [ElasticSearch](https://elastic.co) cluster. It acts as a reverse proxy, routing requests from clients to services. Arc is extended through plugins, which provide extra functionality and services beyond the ElasticSearch's RESTful API. It can perform various cross-cutting tasks such as basic authentication, logging, rate-limiting, source/referers whitelisting, analytics etc. These functionalities can clearly be extended by adding a plugin encapsulating a desired functionality. It also provides some useful abstractions that helps in managing and controlling the access
44
to ElasticSearch's RESTful API.
55

66
## Table of contents
@@ -14,10 +14,10 @@ to ElasticSearch's RESTful API.
1414

1515
## Overview
1616

17-
When Arc is deployed, every client request being made to the Elasticsearch
18-
will hit Arc first and then be proxied to the Elasticsearch cluster. In between requests and responses, Arc
19-
may execute the installed plugins, essentially extending the Elasticsearch API feature set. Arc effectively
20-
becomes an entry point for every API request made to Elasticsearch. Arc can be used and deployed against any
17+
When Arc is deployed, every client request being made to the Elasticsearch
18+
will hit Arc first and then be proxied to the Elasticsearch cluster. In between requests and responses, Arc
19+
may execute the installed plugins, essentially extending the Elasticsearch API feature set. Arc effectively
20+
becomes an entry point for every API request made to Elasticsearch. Arc can be used and deployed against any
2121
Elasticsearch cluster (locally and hosted as provided by [Appbase.io](https://appbase.io)).
2222

2323
```
@@ -60,12 +60,12 @@ In order to run arc, you'll require an Elasticsearch node. There are multiple wa
6060

6161
2. Start a single node Elasticsearch cluster locally
6262

63-
docker run -d --rm --name elasticsearch -p 9200:9200 -p 9300:9300 --net=arc -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.0
63+
docker run -d --rm --name elasticsearch -p 9200:9200 -p 9300:9300 --net=arc -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:7.2.0
6464

6565
3. Start Arc locally
6666

6767
docker build -t arc . && docker run --rm --name arc -p 8000:8000 --net=arc --env-file=config/docker.env arc
68-
68+
6969
For convenience, the steps described above are combined into a single `docker-compose` file. You can execute the file with command:
7070

7171
docker-compose up
@@ -172,16 +172,16 @@ A `User` grants a `Permission` to a certain `User`, predefining its capabilities
172172

173173
#### Category
174174

175-
Categories can be used to control access to data and APIs in Arc. Along with Elasticsearch APIs, Categories cover the APIs provided by Arc itself to allow fine-grained control over the API consumption. For Elasticsearch, Categories broadly resembles to the API [classification](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) that Elasticsearch
176-
provides such as **Document APIs**, **Search APIs**, **Indices APIs** and so on. For Arc, Categories resembles to the
175+
Categories can be used to control access to data and APIs in Arc. Along with Elasticsearch APIs, Categories cover the APIs provided by Arc itself to allow fine-grained control over the API consumption. For Elasticsearch, Categories broadly resembles to the API [classification](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) that Elasticsearch
176+
provides such as **Document APIs**, **Search APIs**, **Indices APIs** and so on. For Arc, Categories resembles to the
177177
additional APIs on top of Elasticsearch APIs, such as analytics and book keeping. Refer to category [docs](https://github.com/appbaseio/arc/blob/ugo/update-readme/31-12-2018/docs/categories.md) for the list of
178178
categories that Arc supports.
179179

180180
#### ACL
181181

182182
ACLs allow a fine grained control over the Elasticsearch APIs in addition to the Categories. Each ACL resembles an
183-
action performed by an Elasticsearch API. For brevity, setting and organising Categories automatically sets the default
184-
ACLs associated with the set Categories. Setting ACLs adds just another level of control to provide access to
183+
action performed by an Elasticsearch API. For brevity, setting and organising Categories automatically sets the default
184+
ACLs associated with the set Categories. Setting ACLs adds just another level of control to provide access to
185185
Elasticsearch APIs within a given Category. Refer to acl [docs](https://github.com/appbaseio/arc/blob/ugo/update-readme/31-12-2018/docs/acls.md) for the list of acls that Arc supports.
186186

187187
#### Op
@@ -194,8 +194,8 @@ of the plugin. Operation is currently classified into three kinds:
194194
- `Write`: operation permits write requests exclusively.
195195
- `Delete`: operation permits delete requests exclusively.
196196

197-
In order to allow a user or permission to make requests that involve modifying the data, a combination of the above
198-
operations would be required. For example: `["read", "write"]` operation would allow a user or permission to perform
197+
In order to allow a user or permission to make requests that involve modifying the data, a combination of the above
198+
operations would be required. For example: `["read", "write"]` operation would allow a user or permission to perform
199199
both read and write requests but would forbid making delete requests.
200200

201201
#### Request Logging

config/manual-http-jwt.env

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ ES_CLUSTER_URL=http://127.0.0.1:9200
33
USERNAME=foo
44
PASSWORD=bar
55

6-
JWT_RSA_PUBLIC_KEY_DEST=http://localhost:8500/rsa-public
76
HTTPS_CERT=sample/server.crt
87
HTTPS_KEY=sample/server.key
98
JWT_ROLE_KEY=role

docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
version: '3'
22
services:
33
elasticsearch:
4-
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.0
4+
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.2.0
55
container_name: elasticsearch
66
networks:
77
- arc

go.mod

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,22 @@ module github.com/appbaseio/arc
22

33
require (
44
github.com/dgrijalva/jwt-go v3.2.0+incompatible
5-
github.com/fortytw2/leaktest v1.2.0 // indirect
65
github.com/gobuffalo/envy v1.6.15 // indirect
76
github.com/gobuffalo/packr v1.22.0
8-
github.com/google/go-cmp v0.2.0 // indirect
97
github.com/google/uuid v1.0.0
10-
github.com/gorilla/context v1.1.1 // indirect
11-
github.com/gorilla/mux v1.6.2
12-
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 // indirect
13-
github.com/olivere/elastic v6.2.16+incompatible
8+
github.com/gorilla/mux v1.7.1
9+
github.com/hashicorp/go-hclog v0.10.0 // indirect
10+
github.com/hashicorp/go-retryablehttp v0.6.3
11+
github.com/olivere/elastic v6.2.21+incompatible
12+
github.com/olivere/elastic/v7 v7.0.4
13+
github.com/robfig/cron v1.1.0
1414
github.com/rogpeppe/go-internal v1.2.2 // indirect
1515
github.com/rs/cors v1.6.0
1616
github.com/siddharthlatest/mustache v0.0.0-20160118163553-00029677272d
17+
github.com/smartystreets/goconvey v1.6.4
1718
github.com/stretchr/testify v1.3.0
1819
github.com/ulule/limiter v2.2.0+incompatible
19-
golang.org/x/crypto v0.0.0-20190103213133-ff983b9c42bc
20+
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
2021
gopkg.in/natefinch/lumberjack.v2 v2.0.0
21-
gopkg.in/olivere/elastic.v6 v6.2.21
22+
gopkg.in/olivere/elastic.v6 v6.2.26
2223
)

0 commit comments

Comments
 (0)