Merge branch 'master' into revert-484-revert-482-fix/queue-parsing

Author: ztsorojev

.circleci/config.yml

@@ -1,47 +1,37 @@
 version: 2.1
 orbs:
-  datacamp-ecr: datacamp/ecr@0
-  datacamp-deploy: datacamp/deploy@1
+  datacamp-ecr: datacamp/ecr@1
+  datacamp-deploy: datacamp/deploy@2
 workflows:
   version: 2
-  build_test_deploy:
+  build-and-deploy:
     jobs:
       - datacamp-ecr/build_and_push_image_to_ecr:
           name: build
           context: org-global
-          aws-access-key-id: $STAGING_AWS_ACCESS_KEY_ID
-          aws-secret-access-key: $STAGING_AWS_SECRET_ACCESS_KEY
-          account-url: $STAGING_ECR_URL
-          extra-docker-args: "--build-arg NPM_TOKEN=${NPM_TOKEN}"
-          repo: rdoc-app
+          aws-access-key-id: $OPS_AWS_ACCESS_KEY_ID
+          aws-secret-access-key: $OPS_AWS_SECRET_ACCESS_KEY
+          account-url: $OPS_ECR_URL
+          puller-account-ids: '["301258414863", "487088987264"]'
       - datacamp-deploy/deploy: # Staging
           context: org-global
           requires:
             - build
-          new-deploy-opt-in: true
-          deploy-url: $STAGING_LAMBDA_DEPLOY_URL
-          deploy-password: $STAGING_LAMBDA_DEPLOY_PASSWORD
+          environment: staging
+          aws-access-key-id: $STAGING_AWS_ACCESS_KEY_ID
+          aws-secret-access-key: $STAGING_AWS_SECRET_ACCESS_KEY
           filters:
             branches:
               only:
                 - master
-      - datacamp-ecr/pull_push_to_account:
-          context: org-global
-          repo: rdoc-app
-          filters:
-            tags:
-              only: /^release-.*/
-            branches:
-              ignore: /.*/
       - datacamp-deploy/deploy: # Production
           context: org-global
           requires:
-            - datacamp-ecr/pull_push_to_account
-          new-deploy-opt-in: true
-          deploy-url: $PROD_LAMBDA_DEPLOY_URL
-          deploy-password: $PROD_LAMBDA_DEPLOY_PASSWORD
+            - build
+          environment: prod
+          aws-access-key-id: $PROD_AWS_ACCESS_KEY_ID
+          aws-secret-access-key: $PROD_AWS_SECRET_ACCESS_KEY
           filters:
-            tags:
-              only: /^release-.*/
             branches:
-              ignore: /.*/
+              only:
+                - master

CONTRIBUTING.md

@@ -1,40 +1,22 @@
-![rdocumentation_site_banner](https://cloud.githubusercontent.com/assets/1741726/17966806/d52c646a-6ac3-11e6-8f61-60379cfd70bb.png)
+# RDocumentation API
 
-rdocumentation.org provides the R community with centralized, quality and easy to search documentation.
-
-R documentation sifts through all CRAN, GitHub and BioConductor packages hourly, parses the documentation files and indexes them in an Elasticsearch database. This makes rdocumentation.org the best online resource to browse all R package documentation.
-
-The RDocs project is completely open-source. This repository contains the source code for the NodeJS web application that serves [www.rdocumentation.org](https://www.rdocumentation.org). For other codebases, you can check out [`CONTRIBUTING.md`](CONTRIBUTING.md).
-
-## Features
-
-- Quick search through all packages and functions
-
-![screen shot 2016-08-25 at 14 07 05](https://cloud.githubusercontent.com/assets/1741726/17968459/41bee176-6acd-11e6-9431-3aec36ffd8c8.png)
-
-- Complete search through all packages and function
-
-![screen shot 2016-08-25 at 14 08 52](https://cloud.githubusercontent.com/assets/1741726/17968498/7ce9a6aa-6acd-11e6-9276-4d5ced4523b3.png)
-
-- Easily assess package quality
+_Important notes:_
 
-![screen shot 2016-08-25 at 14 11 27](https://cloud.githubusercontent.com/assets/1741726/17968583/df47301a-6acd-11e6-9a28-5167b768fbf1.png)
+1. Please read this [confluence page](https://datacamp.atlassian.net/wiki/spaces/PRODENG/pages/2314469377/RDocumentation) which explains the architecture of how RDocumentation works.
+2. This repo is now only used for its API. Please disregard all the UI code here. THe UI now lives in [datacamp/rdocumentation-2.0](https://github.com/datacamp/rdocumentation-2.0).
 
-- Post _community examples_ to help the community understand how to use a function
-
-![screen shot 2016-08-25 at 14 14 37](https://cloud.githubusercontent.com/assets/1741726/17968654/492bb8f2-6ace-11e6-8a64-c620e9e98efa.png)
-
-## What this app does
-
-This application is part the rdocumentation project. This app is responsible for storing the RDocumentation data, and bring it to the users through a web interface (the rdocumentation.org site) or through an api (See https://www.rdocumentation.org/docs/ for the public API)
+rdocumentation.org provides the R community with centralized, quality and easy to search documentation.
 
-## Documentation
+R documentation sifts through all CRAN, GitHub and BioConductor packages hourly, parses the documentation files and indexes them in an Elasticsearch database. This makes rdocumentation.org the best online resource to browse all R package documentation.
 
-Documentation of the differents endpoints can be found here: http://www.rdocumentation.org/docs/
+## How the API works
 
-## Issue/Feature request
+You can check docs for the API by running the app locally and going to http://localhost:3000/docs/
 
-Please post a new issue at https://github.com/datacamp/rdocumentation-app/issues for any bug that you encounter or a feature that you would like to see in rdocumetation.org.
+1. Newly parsed packages are added to the `rdocs-app-worker` SQS queue every hour.
+2. That queue is configured to hit the `/task` path which calls the `processMessage` method of the WorkerController
+3. That `processMessage` method adds topics to the mysql database
+4. After a query is sent to the API to request a topic, that topic is stored in Redis so that it's returned faster next time.
 
 ## Development
 
@@ -43,13 +25,16 @@ Please post a new issue at https://github.com/datacamp/rdocumentation-app/issues
 You'll need docker and docker-compose to run this stack locally
 
 - Copy the .env.sample to .env and change relevant variables
+  - If you already have a mysql db running on port 3306, update the `DATABASE_PORT` to another value as well as the port mapping in docker-compose.yml (e.g. change it to `"3308:3306"`)
 - `docker-compose create` to create the redis and mysql container
 - `docker-compose start` to fire up a local redis an mysql
 - Use the same node version in your terminal as the Dockerfile is using: `nvm use 8.16`
 - `npm install`
 - Run the database migrations by doing `npm run migrate`
 - `npm run start-dev`
 
+Once the db is running, you can use a mysql client like dbeaver to access it. Connect to it based on the environment variables you have in `docker-compose.yml`. The server host should just be `localhost`.
+
 ### Troubleshooting
 
 If you get an error: `SequelizeConnectionError: ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client`
@@ -75,15 +60,6 @@ This application runs on the DataCamp infrastructure. Our custom CI flow will:
 - Upload it the ECR
 - Deploy the new version to ECS
 
-## How to contribute
-
-We welcome any contributions that could improves rdocumentation.org. There is multiple ways of contributing:
-
-- Report when some packages are missing/outdated/incorrect by creating an issue.
-- Report bugs.
-- Help us improves by proposing features.
-- Directly contribute by forking the repo and making changes.
-
 ## License
 
 See the [LICENSE](LICENSE.md) file for license rights and limitations (MIT).

README.md

@@ -0,0 +1,40 @@
+The information on this page comes from the original Readme when this app was used for its UI as well, but since the UI now lives in [datacamp/rdocumentation-2.0](https://github.com/datacamp/rdocumentation-2.0), this Readme is just there to remind you the good old' days of RDocumentation.
+
+## Features
+
+- Quick search through all packages and functions
+
+![screen shot 2016-08-25 at 14 07 05](https://cloud.githubusercontent.com/assets/1741726/17968459/41bee176-6acd-11e6-9431-3aec36ffd8c8.png)
+
+- Complete search through all packages and function
+
+![screen shot 2016-08-25 at 14 08 52](https://cloud.githubusercontent.com/assets/1741726/17968498/7ce9a6aa-6acd-11e6-9276-4d5ced4523b3.png)
+
+- Easily assess package quality
+
+![screen shot 2016-08-25 at 14 11 27](https://cloud.githubusercontent.com/assets/1741726/17968583/df47301a-6acd-11e6-9a28-5167b768fbf1.png)
+
+- Post _community examples_ to help the community understand how to use a function
+
+![screen shot 2016-08-25 at 14 14 37](https://cloud.githubusercontent.com/assets/1741726/17968654/492bb8f2-6ace-11e6-8a64-c620e9e98efa.png)
+
+## What this app does
+
+This application is part the rdocumentation project. This app is responsible for storing the RDocumentation data, and bring it to the users through a web interface (the rdocumentation.org site) or through an api (See https://www.rdocumentation.org/docs/ for the public API)
+
+## Documentation
+
+Documentation of the differents endpoints can be found here: http://www.rdocumentation.org/docs/
+
+## Issue/Feature request
+
+Please post a new issue at https://github.com/datacamp/rdocumentation-app/issues for any bug that you encounter or a feature that you would like to see in rdocumetation.org.
+
+## How to contribute
+
+We welcome any contributions that could improves rdocumentation.org. There is multiple ways of contributing:
+
+- Report when some packages are missing/outdated/incorrect by creating an issue.
+- Report bugs.
+- Help us improves by proposing features.
+- Directly contribute by forking the repo and making changes.