diff --git a/contributing/release-checklist.md b/contributing/release-checklist.md index e622dd4..b8ab1f3 100644 --- a/contributing/release-checklist.md +++ b/contributing/release-checklist.md @@ -103,7 +103,6 @@ The following pages require manual content review beyond link verification: |------|----------------| | `docs/deploy-nbs7/real-time-reporting.md` | Update the version number in the 2 code blocks to set the environment variables | | `docs/deploy-nbs7/real-time-reporting/rtr-java-services.md` | Review the consolidation warning callout for accuracy. | -| `docs/deploy-nbs7/real-time-reporting/data-compare-tool.md` | The NEDSS-DataCompare link is pinned to `main` — that repo has no version tags. Confirm this is still correct. | | `docs/deploy-nbs7/microservices-deployment/nnd-service/on-prem-data-sync.md` | Confirm the NEDSS-NNDSS release at the new tag includes a `vX.Y.Z.NEDSS.NBS.Modernized.Documentation.zip` asset. If missing, coordinate with the dev team before publishing. | | `docs/deploy-nbs7/microservices-deployment/nnd-service/on-prem-nnd-sync.md` | Same documentation zip caveat as `on-prem-data-sync.md`. | | `docs/maintain-nbs7/eks-upgrade.md` | Update the Kubernetes versions table. Confirm the conditional add-ons step still reflects correct module version behavior. *(Page not yet created as of 7.12.)* | diff --git a/docs/deploy-nbs7/architecture-and-microservices.md b/docs/deploy-nbs7/architecture-and-microservices.md index e86701d..16af55c 100644 --- a/docs/deploy-nbs7/architecture-and-microservices.md +++ b/docs/deploy-nbs7/architecture-and-microservices.md @@ -75,13 +75,7 @@ Real-Time Reporting (RTR) provides rapid transformation and delivery of data fro The following services comprise RTR: -- **Liquibase job**: Database version control management that deploys stored procedures, tables and views required for RTR pipeline -- **Person service**: Processes Patient and Provider change events -- **Observation services**: Processes Observation change events -- **Organization service**: Processes Organization change events -- **LDF service**: Processes LDF or State-defined field data change events -- **Investigation service**: Processes Public_health_case change events and provides information for page builder investigation, notifications, confirmation method, and updates the `PublicHealthCaseFact_Modern` datamart -- **Post-processing service**: Calls the post-processing stored procedures to hydrate dimensions, fact tables, and datamarts +- **Reporting Pipeline service**: ⏺ consumes ODSE/SRTE change events from Kafka, transforms them via stored procedures, and hydrates the reporting database's dimensions, facts, and datamarts in near real time. - **Debezium service**: Monitors and streams the selected list of `NBS_ODSE` and `NBS_SRTE` tables to Kafka topics - **Kafka sink service**: Persists the data from the Kafka topics to the `RDB_Modern` database tables diff --git a/docs/deploy-nbs7/deploy-on-aws/provision-aws.md b/docs/deploy-nbs7/deploy-on-aws/provision-aws.md index 5305b6b..fd37eec 100644 --- a/docs/deploy-nbs7/deploy-on-aws/provision-aws.md +++ b/docs/deploy-nbs7/deploy-on-aws/provision-aws.md @@ -42,16 +42,6 @@ Complete these steps to download the infrastructure package and prepare your env 1. Update the `terraform.tfvars` and `terraform.tf` with your environment-specific values. - If you plan to deploy the Data Compare tool, also add the following variables to your `terraform.tfvars` before running `terraform apply`. If you deployed to a non-default namespace, you should adjust the `datacompare_namespace_and_service` value accordingly. - - ```hcl - create_datacompare_irsa = true - datacompare_s3_bucket_name = "" - datacompare_s3_bucket_keyname_prefix = "" - datacompare_namespace_and_service = ["default:data-compare-api-service", "default:data-compare-processor-service"] - ``` - - This creates the IAM role (`-datacompare-role`) and S3 access policy required by the Data Compare pods. The role ARN is referenced during [Data Compare deployment](../../deploy-nbs7/real-time-reporting/data-compare-tool.html). ## Validate AWS access and network prerequisites diff --git a/docs/deploy-nbs7/real-time-reporting/data-compare-tool.md b/docs/deploy-nbs7/real-time-reporting/data-compare-tool.md deleted file mode 100644 index 436da41..0000000 --- a/docs/deploy-nbs7/real-time-reporting/data-compare-tool.md +++ /dev/null @@ -1,211 +0,0 @@ ---- -title: Data Compare -layout: page -parent: Deploy real-time reporting -nav_order: 6 -description: Explains deployment and use of Data Compare to validate RTR output against classic ETL results. -redirect_from: - - /docs/7_feature_preview/6_data_compare_tool.html - - /docs/7_feature_preview/6_data_compare_tool/ ---- - -# Deploy and use the RTR Data Compare validation tool - -This page covers deploying and using the Data Compare tool to validate RTR output against classic ETL results. - -## On this page -{: .no_toc .text-delta } - -1. TOC -{:toc} - -## Overview - -The Data Compare tool is an optional RTR validation service that allows STLT users to compare data processed by RTR against the classic ETL pipeline and identify differences. - -> This service is optional. STLTs can choose to install it only if they require RTR validation capabilities. -{: .note } - -The tool consists of two containerized services that communicate asynchronously through Kafka: - -- **Data Compare API** - pulls and prepares data from designated tables, then uploads it to a cloud storage bucket -- **Data Compare Processor** - retrieves data from the cloud storage bucket and performs the comparison logic - -Database changes are managed by Liquibase, integrated within the `DataCompareAPI` service. Schema changes are applied automatically during deployment. The database objects in the following directory are for reference only: [NEDSS-DataCompare/DataCompareAPIs/.../db/data_internal](https://github.com/CDCgov/NEDSS-DataCompare/tree/main/DataCompareAPIs/src/main/resources/db/data_internal) - -## Prerequisites - -Before deploying the Data Compare tool, verify the following: - -- Access to a cloud storage bucket for data exchange between the API and Processor services. These steps currently use Amazon S3. If you are not using Amazon S3, consult your cloud administrator for equivalent storage configuration. - - For AWS deployments, an IAM role granting the Data Compare pods access to S3 must be provisioned before deployment. This role is created by Terraform in NEDSS-Infrastructure when `create_datacompare_irsa = true` is set in your `terraform.tfvars`. The role is named `-datacompare-role`. See [Provision the AWS environment](../deploy-on-aws/provision-aws.html) for details. - -- Keycloak configured with the Data Compare API profile: [NEDSS-Helm/charts/keycloak/extra][nedss-helm-keycloak-extra] - -If your Keycloak pod uses a name or namespace other than the default, update the `authUri` in your `values.yaml`: - -```yaml -authUri: "http://keycloak.default.svc.cluster.local/auth/realms/NBS" -``` - -> These steps require a Unix-compatible shell. On Windows, use Git Bash, WSL, or an equivalent terminal emulator. -{: .note } -> Verify that you are connected to the correct Kubernetes cluster before proceeding. To confirm, run `kubectl config current-context`. -{: .important } - -## Deploy the Data Compare API - -Follow these steps to configure and deploy the Data Compare API Helm chart. - -The Helm chart is located in `charts/data-compare-api-service` in the [NEDSS-Helm repository][nedss-helm-data-compare-api-chart]. - -1. Configure `values.yaml`. Replace all placeholder values before installation: - - ```yaml - image: - # Data Compare API image - repository: "quay.io/us-cdcgov/cdc-nbs-modernization/data-compare-api-service" - pullPolicy: IfNotPresent - # Replace with the target release version tag, e.g. v1.0.1 - tag: - - ingressHost: "data.EXAMPLE_DOMAIN" - jdbc: - # SQL Server endpoint - dbserver: "EXAMPLE_DB_ENDPOINT" - username: "EXAMPLE_ODSE_DB_USER" - password: "EXAMPLE_ODSE_DB_USER_PASSWORD" - authUri: "http://keycloak.default.svc.cluster.local/auth/realms/NBS" - s3: - # AWS-specific: replace with your AWS Region and S3 bucket name - region: "AWS REGION" - bucketName: "S3 BucketName" - ``` - -1. For AWS deployments, add the IRSA role ARN to `values.yaml` to grant the pod access to S3: - - ```yaml - serviceAccount: - annotations: - eks.amazonaws.com/role-arn: "arn:aws:iam:::role/-datacompare-role" - ``` - - Replace `` and `` with your values. The role is created during infrastructure provisioning. See [Provision the AWS environment](../deploy-on-aws/provision-aws.html) for details. - -1. Install the Helm chart: - - ```bash - helm install data-compare-api-service -f ./data-compare-api-service/values.yaml data-compare-api-service - ``` - -1. Verify the pod is running: - - ```bash - kubectl get pods - ``` - -1. Validate the service by opening the Swagger UI. Replace `` with your actual domain: - - ```text - https:///comparison/swagger-ui/index.html - ``` - -## Deploy the Data Compare Processor - -Follow these steps to configure and deploy the Data Compare Processor Helm chart. - -The Helm chart is located in `charts/data-compare-processor-service` in the [NEDSS-Helm repository][nedss-helm-data-compare-processor-chart]. - -> The Processor is a Kafka consumer microservice and does not expose any API endpoints. -{: .note } - -1. Configure `values.yaml`. Replace all placeholder values before installation: - - ```yaml - image: - # Data Compare Processor image - repository: "quay.io/us-cdcgov/cdc-nbs-modernization/data-compare-processor-service" - pullPolicy: IfNotPresent - # Replace with the target release version tag, e.g. v1.0.1 - tag: - - ingressHost: "data.EXAMPLE_DOMAIN" - jdbc: - # SQL Server endpoint - dbserver: "EXAMPLE_DB_ENDPOINT" - username: "EXAMPLE_ODSE_DB_USER" - password: "EXAMPLE_ODSE_DB_USER_PASSWORD" - authUri: "http://keycloak.default.svc.cluster.local/auth/realms/NBS" - s3: - # AWS-specific: replace with your AWS Region and S3 bucket name - region: "AWS REGION" - bucketName: "S3 BucketName" - ``` - -1. For AWS deployments, add the IRSA role ARN to `values.yaml` to grant the pod access to S3: - - ```yaml - serviceAccount: - annotations: - eks.amazonaws.com/role-arn: "arn:aws:iam:::role/-datacompare-role" - ``` - - Replace `` and `` with your values. The role is created during infrastructure provisioning. See [Provision the AWS environment](../deploy-on-aws/provision-aws.html) for details. - -1. Install the Helm chart: - - ```bash - helm install data-compare-processor-service -f ./data-compare-processor-service/values.yaml data-compare-processor-service - ``` - -1. Verify the pod is running: - - ```bash - kubectl get pods - ``` - -## Configure ingress - -The Data Compare API uses the same ingress as the data ingestion service. Reuse the ingress configuration as needed: [dataingestion-service/templates/ingress.yaml](https://github.com/CDCgov/NEDSS-Helm/blob/10623c0d9788a6513bd51f4b6ed4eb0f79b30a2f/charts/dataingestion-service/templates/ingress.yaml#L99) - -## Verify the deployment - -Confirm both services are running without errors: - -```bash -kubectl get pods -kubectl logs -``` - -Both services are ready when they are healthy and the Processor begins consuming from Kafka. - -## Use the Data Compare tool - -The comparison process relies on the `Data_Compare_Config` table, which Liquibase creates and populates when the Data Compare API deploys. The table comes preloaded with records containing table names and queries that determine what data to compare. - -To start a comparison, call: - -```text -POST /comparison/api/data-compare -``` - -Pass the `runNowMode` header to control scope: - -- `true` - runs only on records in the config table where `runNow = true`; resets `runNow` to `false` when complete -- `false` - runs on all records in the config table - -This is an asynchronous endpoint. If authentication passes and there are no logical errors, it returns a success response immediately. The comparison runs in the background. - -The following diagram shows the end-to-end data flow: - -```text -API → Pull data from SQL table → Upload to S3 → Kafka → Processor → Pull from S3 → Perform comparison → Upload results to S3 -``` - -> This data flow uses Amazon S3 as the storage provider. If you are not using Amazon S3, the upload and retrieval steps differ based on your cloud provider. -{: .note } - -[nedss-helm-keycloak-extra]: -[nedss-helm-data-compare-api-chart]: -[nedss-helm-data-compare-processor-chart]: diff --git a/docs/deploy-nbs7/real-time-reporting/debezium.md b/docs/deploy-nbs7/real-time-reporting/debezium.md index 2c0cf59..3ed54c0 100644 --- a/docs/deploy-nbs7/real-time-reporting/debezium.md +++ b/docs/deploy-nbs7/real-time-reporting/debezium.md @@ -31,7 +31,7 @@ Follow these steps to configure and deploy the Debezium Helm chart for RTR. 1. Locate the Debezium Helm chart in the [NEDSS-Helm repository][nedss-helm-debezium-chart]. -1. Configure `values.yaml`. Replace all placeholder values before installation. +1. Edit the "EXAMPLE_*" placeholder values in `values.yaml` and replace with the correct values for your environment (eg EXAMPLE_KAFKA_ENDPOINT). To retrieve your Kafka bootstrap server endpoints, see [Get bootstrap brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-get-bootstrap-brokers.html) in the AWS MSK documentation. @@ -44,51 +44,12 @@ Follow these steps to configure and deploy the Debezium Helm chart for RTR. properties: # Kafka bootstrap server endpoint from AWS MSK - bootstrap_server: "EXAMPLE_MSK_KAFKA_ENDPOINT" - - # Connector for NBS_ODSE transactional data - sqlserverconnector_odse: - config: - database.hostname: "nbs-db.private-EXAMPLE_DOMAIN" - database.port: "1433" - database.user: "EXAMPLE_ODSE_DB_USER" - database.password: "EXAMPLE_ODSE_DB_USER_PASSWORD" - database.dbname: "nbs_odse" - database.names: "nbs_odse" - database.server.name: "odse" - database.history.kafka.bootstrap.servers: "EXAMPLE_MSK_KAFKA_ENDPOINT" - schema.history.internal.kafka.bootstrap.servers: "EXAMPLE_MSK_KAFKA_ENDPOINT" - - # Connector for NBS_ODSE metadata - sqlserverconnector_odse_meta: - config: - database.hostname: "nbs-db.private-EXAMPLE_DOMAIN" - database.port: "1433" - database.user: "EXAMPLE_ODSE_DB_USER" - database.password: "EXAMPLE_ODSE_DB_USER_PASSWORD" - database.dbname: "nbs_odse" - database.names: "nbs_odse" - database.server.name: "odse-meta" - database.history.kafka.bootstrap.servers: "EXAMPLE_MSK_KAFKA_ENDPOINT" - schema.history.internal.kafka.bootstrap.servers: "EXAMPLE_MSK_KAFKA_ENDPOINT" - - # Connector for NBS_SRTE reference and terminology data - sqlserverconnector_srte: - config: - database.hostname: "nbs-db.private-EXAMPLE_DOMAIN" - database.port: "1433" - database.user: "EXAMPLE_SRTE_DB_USER" - database.password: "EXAMPLE_SRTE_DB_USER_PASSWORD" - database.dbname: "nbs_srte" - database.names: "nbs_srte" - database.server.name: "srte" - database.history.kafka.bootstrap.servers: "EXAMPLE_MSK_KAFKA_ENDPOINT" - schema.history.internal.kafka.bootstrap.servers: "EXAMPLE_MSK_KAFKA_ENDPOINT" + bootstrap_server: "EXAMPLE_KAFKA_ENDPOINT" env: # Kafka bootstrap server endpoint from AWS MSK - name: BOOTSTRAP_SERVERS - value: "EXAMPLE_MSK_KAFKA_ENDPOINT" + value: "EXAMPLE_KAFKA_ENDPOINT" ``` 1. Install the pod: @@ -100,7 +61,7 @@ Follow these steps to configure and deploy the Debezium Helm chart for RTR. 1. Verify the pod is running: ```bash - kubectl get pods + kubectl get deployment debezium-debezium-rtr-connect ``` 1. Validate the service. @@ -119,7 +80,7 @@ If issues persist after initial troubleshooting, contact support at diff --git a/docs/deploy-nbs7/real-time-reporting/kafka-connector.md b/docs/deploy-nbs7/real-time-reporting/kafka-connector.md index 2561ff4..fb73cb3 100644 --- a/docs/deploy-nbs7/real-time-reporting/kafka-connector.md +++ b/docs/deploy-nbs7/real-time-reporting/kafka-connector.md @@ -42,14 +42,6 @@ Follow these steps to configure and deploy the Kafka connector Helm chart for RT # Replace with the target release version tag, e.g. v1.0.1 tag: - sqlServerConnector: - config: - # [SME REVIEW: confirm databaseName should be rdb_modern, not rdb] - connection.url: "jdbc:sqlserver://nbs-db.EXAMPLE_FIXME.nbspreview.com:1433;databaseName=rdb;encrypt=true;trustServerCertificate=true" - # SQL Server username for reporting database - connection.user: "EXAMPLE_FIXME" - connection.password: "EXAMPLE_FIXME" - kafka: # Kafka bootstrap server endpoint from AWS MSK bootstrapServers: "EXAMPLE_FIXME" @@ -64,7 +56,7 @@ Follow these steps to configure and deploy the Kafka connector Helm chart for RT 1. Verify the pod is running: ```bash - kubectl get pods + kubectl get deployment kafka-connect-sink-cp-kafka-connect ``` 1. Validate the service. diff --git a/docs/deploy-nbs7/real-time-reporting/liquibase.md b/docs/deploy-nbs7/real-time-reporting/liquibase.md deleted file mode 100644 index b7e60dc..0000000 --- a/docs/deploy-nbs7/real-time-reporting/liquibase.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -title: Liquibase -layout: page -parent: Deploy real-time reporting -nav_order: 1 -description: Explains deployment of the RTR Liquibase job to apply required schema changes across NBS databases. -redirect_from: - - /docs/7_feature_preview/1_liquibase.html - - /docs/7_feature_preview/1_liquibase/ ---- - -# Deploy Liquibase for real-time reporting (RTR) - -In RTR deployments, the Liquibase job runs once and applies required SQL Server database changes for onboarding and upgrades. This page covers steps to deploy the Liquibase Helm chart. - -## On this page -{: .no_toc .text-delta } - -1. TOC -{:toc} - -## Installing Liquibase - -Follow these steps to configure and deploy the Liquibase Helm chart for RTR. - -> Verify that you are connected to the correct Kubernetes cluster before proceeding. To confirm, run `kubectl config current-context`. -{: .important } - -1. Locate the Liquibase Helm chart in the [NEDSS-Helm repository][nedss-helm-liquibase-chart]. -1. Configure `values.yaml`. Replace all placeholder values before installation: - - Replace `app.EXAMPLE_DOMAIN` with the URL of your modern app (see [Deploy Traefik ingress controller](../../deploy-nbs7/initial-kubernetes-deployment/initial-kubernetes-deployment.html#deploy-traefik-ingress-controller)): - - ```yaml - image: - repository: "quay.io/us-cdcgov/cdc-nbs-modernization/liquibase-service" - tag: # Example: v1.0.1 - ``` - - Replace `EXAMPLE_DB_ENDPOINT` with your SQL Server endpoint and set credentials. - For `username`, use `db_deploy_admin`. - - To retrieve your Kafka bootstrap server endpoints, see [Get bootstrap brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-get-bootstrap-brokers.html) in the AWS MSK documentation. - - The following fields define the JDBC connection strings for each NBS database. Each URL - identifies the SQL Server endpoint, the database name, and the connection security settings. - - ```yaml - jdbc: - # SQL Server system database - master_db_url: "jdbc:sqlserver://EXAMPLE_DB_ENDPOINT:1433;databaseName=master;integratedSecurity=false;encrypt=true;trustServerCertificate=true" - # NBS transactional database - odse_db_url: "jdbc:sqlserver://EXAMPLE_DB_ENDPOINT:1433;databaseName=nbs_odse;integratedSecurity=false;encrypt=true;trustServerCertificate=true" - # NBS reference and terminology database - srte_db_url: "jdbc:sqlserver://EXAMPLE_DB_ENDPOINT:1433;databaseName=nbs_srte;integratedSecurity=false;encrypt=true;trustServerCertificate=true" - # Legacy reporting database (RDB path) - rdb_db_url: "jdbc:sqlserver://EXAMPLE_DB_ENDPOINT:1433;databaseName=rdb;integratedSecurity=false;encrypt=true;trustServerCertificate=true" - # Separate reporting database (rdb_modern path only) - rdb_modern_db_url: "jdbc:sqlserver://EXAMPLE_DB_ENDPOINT:1433;databaseName=rdb_modern;integratedSecurity=false;encrypt=true;trustServerCertificate=true" - # db_deploy_admin user created in prerequisites - username: "EXAMPLE_DB_USER" - password: "EXAMPLE_DB_USER_PASSWORD" - # Separate credentials for NBS_SRTE if applicable - srte_username: "EXAMPLE_SRTE_DB_USER" - srte_password: "EXAMPLE_SRTE_DB_USER_PASSWORD" - ``` - -1. Install the pod: - - ```bash - helm install -f ./liquibase/values.yaml liquibase ./liquibase/ - ``` - -1. Verify the pod is running: - - ```bash - kubectl get pods - ``` - -1. Validate Liquibase updates from NBS databases using the `DATABASECHANGELOG` table: - - ```sql - --Last script executed should be 999-_database_object_permission_grants-001.sql. - USE NBS_ODSE; - SELECT TOP 1 * - FROM NBS_ODSE.DBO.DATABASECHANGELOG - ORDER BY DATEEXECUTED DESC; - - USE NBS_SRTE; - SELECT TOP 1 * - FROM NBS_SRTE.DBO.DATABASECHANGELOG - ORDER BY DATEEXECUTED DESC; - - USE RDB; - SELECT TOP 1 * - FROM RDB.DBO.DATABASECHANGELOG - ORDER BY DATEEXECUTED DESC; - - USE RDB_MODERN; - SELECT TOP 1 * - FROM RDB_MODERN.DBO.DATABASECHANGELOG - ORDER BY DATEEXECUTED DESC; - ``` - -If validation fails or returns unexpected results, see [Troubleshooting Liquibase installation](#troubleshooting-liquibase-installation). - -> If you arrived here from [Create service users and database objects](../real-time-reporting/#create-service-users-and-database-objects), return there to continue after Liquibase completes successfully. -{: .note } - -## Troubleshooting Liquibase installation - -Troubleshooting can vary by database. If issues persist after initial troubleshooting, contact support at . - -### User permission errors - -If a Liquibase execution fails due to a user permission error, run the following script: - -```sql -USE [NBS_SRTE] -GO -ALTER USER [nbs_ods] WITH DEFAULT_SCHEMA=[dbo] -GO -USE [NBS_SRTE] -GO -ALTER ROLE [db_owner] ADD MEMBER [nbs_ods] -``` - -### Migration errors - -If you see `Migration failed` or `Invalid object name` errors, run the following script: - -```sql -Use rdb_modern; -IF NOT EXISTS (SELECT 1 FROM sysobjects WHERE name = 'nrt_odse_Page_cond_mapping' and xtype = 'U') - BEGIN - CREATE TABLE [dbo].[nrt_odse_Page_cond_mapping] ( - [page_cond_mapping_uid] [bigint] NOT NULL, - [wa_template_uid] [bigint] NOT NULL, - [condition_cd] [varchar](20) NOT NULL, - [add_time] [datetime] NOT NULL, - [add_user_id] [bigint] NOT NULL, - [last_chg_time] [datetime] NOT NULL, - [last_chg_user_id] [bigint] NOT NULL, - CONSTRAINT [PK_nrt_odse_Page_cond_mapping] PRIMARY KEY CLUSTERED ( - [page_cond_mapping_uid] ASC - ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] - ) ON [PRIMARY]; - END; - -USE RDB_MODERN; -IF NOT EXISTS (SELECT 1 FROM sysobjects WHERE name = 'nrt_odse_NBS_page' and xtype = 'U') - BEGIN - CREATE TABLE [dbo].[nrt_odse_NBS_page] ( - [nbs_page_uid] [bigint] NOT NULL, - [wa_template_uid] [bigint] NOT NULL, - [form_cd] [varchar](50) NULL, - [desc_txt] [varchar](2000) NULL, - [jsp_payload] [image] NULL, - [datamart_nm] [varchar](21) NULL, - [local_id] [varchar](50) NULL, - [bus_obj_type] [varchar](50) NOT NULL, - [last_chg_user_id] [bigint] NOT NULL, - [last_chg_time] [datetime] NOT NULL, - [record_status_cd] [varchar](20) NOT NULL, - [record_status_time] [datetime] NOT NULL, - CONSTRAINT [PK_nrt_odse_NBS_page] PRIMARY KEY CLUSTERED ( - [nbs_page_uid] ASC - ) - WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] - ) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]; - END; - -USE RDB_MODERN; -IF NOT EXISTS (SELECT 1 FROM sysobjects WHERE name = 'nrt_odse_NBS_rdb_metadata' and xtype = 'U') - BEGIN - CREATE TABLE [dbo].[nrt_odse_NBS_rdb_metadata] ( - [nbs_rdb_metadata_uid] [bigint] NOT NULL, - [nbs_page_uid] [bigint] NULL, - [nbs_ui_metadata_uid] [bigint] NOT NULL, - [rdb_table_nm] [varchar](30) NULL, - [user_defined_column_nm] [varchar](30) NULL, - [record_status_cd] [varchar](20) NOT NULL, - [record_status_time] [datetime] NOT NULL, - [last_chg_user_id] [bigint] NOT NULL, - [last_chg_time] [datetime] NOT NULL, - [local_id] [varchar](50) NULL, - [rpt_admin_column_nm] [varchar](50) NULL, - [rdb_column_nm] [varchar](30) NULL, - [block_pivot_nbr] [int] NULL, - CONSTRAINT [PK_nrt_odse_NBS_rdb_metadata] PRIMARY KEY CLUSTERED ( - [nbs_rdb_metadata_uid] ASC - ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] - ) ON [PRIMARY]; - - CREATE NONCLUSTERED INDEX [RDB_PERF_RDB_TBL_NM] ON [dbo].[nrt_odse_NBS_rdb_metadata] ( - [rdb_table_nm] ASC - ) - INCLUDE ( - [nbs_ui_metadata_uid], - [rdb_column_nm] - ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) - ON [PRIMARY]; - - CREATE NONCLUSTERED INDEX [RDB_PERF_UID_RDB_TBL_NM] ON [dbo].[nrt_odse_NBS_rdb_metadata] ( - [nbs_ui_metadata_uid] ASC, - [rdb_table_nm] ASC - ) - INCLUDE ( - [rdb_column_nm] - ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) - ON [PRIMARY]; - - CREATE NONCLUSTERED INDEX [RDB_PERF_UID] ON [dbo].[nrt_odse_NBS_rdb_metadata]( - [nbs_ui_metadata_uid] ASC - ) - INCLUDE ( - [rdb_column_nm] - ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) - ON [PRIMARY]; - END; -``` - -### Incomplete migration - -If the expected values are not returned for the `NBS_ODSE`, `RDB`, `NBS_SRTE` or `rdb_modern` database and the update is incomplete, clear the `DATABASECHANGELOG` table. Then rerun Liquibase: - -```sql -USE NBS_ODSE; -DELETE FROM NBS_ODSE.dbo.DATABASECHANGELOG; - -USE NBS_SRTE; -DELETE FROM NBS_SRTE.dbo.DATABASECHANGELOG; - -USE RDB; -DELETE FROM RDB.dbo.DATABASECHANGELOG; - -USE rdb_modern; -DELETE FROM rdb_modern.dbo.DATABASECHANGELOG; -``` - -[nedss-helm-liquibase-chart]: diff --git a/docs/deploy-nbs7/real-time-reporting/real-time-reporting.md b/docs/deploy-nbs7/real-time-reporting/real-time-reporting.md index d16986a..7be2fe7 100644 --- a/docs/deploy-nbs7/real-time-reporting/real-time-reporting.md +++ b/docs/deploy-nbs7/real-time-reporting/real-time-reporting.md @@ -47,160 +47,45 @@ To reduce risk, consider setting up RTR in a testing environment before moving t FROM NBS_ODSE.dbo.NBS_Release; ``` - Or: - - ```sql - USE NBS_ODSE; - SELECT * FROM NBS_Configuration WHERE config_key = 'CODE_BASE'; - ``` - -1. Verify that the following classic ETL batch jobs have completed successfully: - - `MasterETL.bat` +1. Run the ETL jobs one final time and make sure they complete successfully. - `PHCMartETL.bat` + - `MasterETL.bat` - `covid19ETL.bat` +1. Choose a reporting database. RTR can write to your existing `RDB` database, or to a new database you create by duplicating `RDB`. Pick one option and use it consistently throughout this guide. + > Back up the `RDB` database before you proceed. This step cannot be undone. {: .warning } -1. Choose a database path and use it consistently throughout this guide. Both paths support Liquibase or script-based installation. - - - **RDB path:** Use `RDB` as the default reporting database. Turn off the classic ETL batch jobs and proceed to the next step. MasterETL remains available for manual recovery runs if needed. - - **rdb_modern path:** Create a separate reporting database. To create the database, see [Set up the rdb_modern database](#set-up-the-rdb_modern-database-rdb_modern-path-only) before moving on to [Create service users and database objects](#create-service-users-and-database-objects). - -1. Set the environment variable for your chosen path. - - - **RDB path:** Insert the following value into `NBS_Configuration`: - - ```sql - IF NOT EXISTS(SELECT 1 FROM NBS_ODSE.DBO.NBS_Configuration WHERE config_key ='ENV' AND config_value ='PROD') - INSERT INTO NBS_ODSE.dbo.NBS_Configuration - (config_key, config_value, short_name, desc_txt, default_value, valid_values, category, add_release, version_ctrl_nbr, add_user_id, add_time, last_chg_user_id, last_chg_time, status_cd, status_time, admin_comment, system_usage, config_value_large) - VALUES(N'ENV', N'PROD', N'RTR reporting database', N'Indicates scripts should be run against RDB database', NULL, N'UAT, PROD', N'RTR', N'7.12.0', 1, 0, getdate(), 0, getdate(), N'A', getdate(), NULL, NULL, NULL); - ``` - - - **rdb_modern path:** This setting overrides the default `RDB` during script execution unless a script explicitly prompts for a database. - - ```sql - IF NOT EXISTS(SELECT 1 FROM NBS_ODSE.DBO.NBS_Configuration WHERE config_key ='ENV' AND config_value ='UAT') - INSERT INTO NBS_ODSE.dbo.NBS_Configuration - (config_key, config_value, short_name, desc_txt, default_value, valid_values, category, add_release, version_ctrl_nbr, add_user_id, add_time, last_chg_user_id, last_chg_time, status_cd, status_time, admin_comment, system_usage, config_value_large) - VALUES(N'ENV', N'UAT', N'RTR reporting database', N'Indicates scripts should be run against UAT rdb_modern database', NULL, N'UAT, PROD', N'RTR', N'7.12.0', 1, 0, getdate(), 0, getdate(), N'A', getdate(), NULL, NULL, NULL); - ``` - -## Set up the rdb_modern database (rdb_modern path only) - -If you are on the **rdb_modern path**, complete this section. If you are on the **RDB path**, move on to [Create service users and database objects](#create-service-users-and-database-objects). For more information on choosing a path, see Step 3 in the [Prerequisites](#prerequisites) section. - -RTR requires a dedicated reporting database. To create `rdb_modern`, you restore a copy of `RDB` under a new name. This keeps the classic ETL-hydrated `RDB` intact and available while `rdb_modern` hosts the data structures the RTR pipeline requires. - -If your `RDB` database is hosted on Amazon RDS, follow the steps in [Back up and restore a SQL Server database on Amazon RDS](../../maintain-nbs7/rds-backup-restore.html). For all other database hosting providers, use your standard backup and restore procedure to create a copy of `RDB` as `rdb_modern`. - -## Create service users and database objects - -Complete the following steps to create the database users, Kubernetes secrets, and database objects that the RTR pipeline requires before [Change Data Capture](#enable-change-data-capture) can be enabled. - -> Generate passwords for each service user before running the scripts. Password generation scripts can take several minutes to run. Do not use spaces, equal signs (`=`), or colons (`:`). These characters cause script execution failures. -{: .important } - -1. **Create admin user.** Run [000-create_rtr_admin_user-001.sql][nedss-datareporting-onboarding-user-scripts] from the NEDSS-DataReporting onboarding user creation scripts. This user provides Liquibase permissions to maintain required database components for RTR and enable Change Data Capture. Review the script and update the `PASSWORD` value before execution. - -1. **Create RTR microservice user logins.** Run [001-service_users_login_creation-001.sql and 002-service_database_user_creation-001.sql][nedss-datareporting-onboarding-user-scripts] from the same directory. These scripts create dedicated user accounts for each RTR microservice, which are referenced in Helm values for RTR services. Review the scripts and update the `PASSWORD` values before execution. + - **Use your existing RDB database** — RTR takes over writing to `RDB`. Turn off the classic ETL batch jobs and proceed to the next step. MasterETL remains available for manual recovery runs if needed. -1. **Create Kubernetes secrets for each service user.** Include the admin user from step 1. Each secret should include the database username and password. Script location: [NEDSS-DataReporting/create-kubernetes-secrets][nedss-helm-k8-secrets-manifest]. For steps to create secrets, see [Create secrets in your cluster](../../deploy-nbs7/initial-kubernetes-deployment/initial-kubernetes-deployment.html#create-secrets-in-your-cluster). + - **Create a new reporting database** (recommended) — Duplicate your existing `RDB`, this lets you run RTR alongside MasterETL to compare results before fully committing. You can use any name for this reporting database, but for the remainder of this guide, we'll refer to it as `RDB_MODERN`. The exact steps for database duplication depend on your SQL Server version and hosting environment. If this process is unfamiliar, see [Microsoft's documentation on backup and restore operations](https://learn.microsoft.com/en-us/sql/relational-databases/backup-restore/back-up-and-restore-of-sql-server-databases?view=sql-server-ver17). -1. **Create required database objects.** Run the scripts for your chosen path: - - The database scripts referenced throughout this guide are maintained in the [NEDSS-DataReporting][nedss-datareporting-liquibase-service] repository. You can create the required database objects through Liquibase, which will automatically implement database schema changes, or you can use the provided scripts to install database schema changes. Both options are referenced in the relevant sections. - {: .important } - - - **Liquibase:** See [Deploy Liquibase](../../deploy-nbs7/real-time-reporting/liquibase.html) to create all necessary objects, then return here to continue. Liquibase also handles future database upgrades automatically, so no manual intervention is needed when you update NBS 7. + > If using a new reporting database, you must use the new reporting execution server to run reports (TODO: add link) + {: .note } - - **Upgrade scripts:** See the script execution sequence and `db_upgrade` script in [NEDSS-DataReporting/db-upgrade][nedss-datareporting-manual-deployment]. Clone or download the repository, then run: +## Create service user - ```bash - upgrade_db.bat - ``` +### Account Permissions - When you update NBS 7 to a new release version, you will need to run the upgrade scripts again. See [After onboarding: database upgrades](#after-onboarding-database-upgrades). +1. **Name**: Any name works, but we recommend using a name descriptive to the role, such as `rtr-service-user`. +1. **Databases Permissions**: + - `NBS_ODSE`: `db_datareader` + - `NBS_SRTE`: `db_datareader` + - `RDB` / `RDB_MODERN`: `db_owner` ## Enable Change Data Capture -Change Data Capture (CDC) streams row-level changes from `NBS_ODSE` and `NBS_SRTE` to Kafka, where RTR services load them into the reporting database. Complete the following steps to load the initial data, enable Change Data Capture, and verify the configuration before deploying RTR services. - -1. **Load data and enable Change Data Capture.** This one-time step is required after all database objects are created. - - - **Liquibase:** The `--load-data` flag is not required when using Liquibase. Proceed to step 2. - - - **Upgrade scripts:** Navigate to [02_onboarding_script_data_load][nedss-datareporting-onboarding-data-load] and clone or download the repository, then run: +[Change Data Capture](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver17) (CDC) streams row-level changes from `NBS_ODSE` and `NBS_SRTE` to Kafka, where RTR services load them into the reporting database. Enabling CDC on these databases does require `sysadmin` level permissions, so be sure to run the bootstrap script with a `sysadmin` account. - ```bash - upgrade_db.bat --load-data master - ``` - -1. **Verify Change Data Capture.** `is_cdc_enabled=1` indicates successful configuration. - - > In the following statements, `cdc` appears as part of SQL Server column and parameter names and refers to **Change Data Capture**, not the Centers for Disease Control and Prevention. - {: .note } - - ```sql - SELECT name, is_cdc_enabled - FROM sys.databases; - - -- View ODSE tables with Change Data Capture enabled - USE NBS_ODSE; - SELECT - name, - CASE WHEN is_tracked_by_cdc = 1 THEN 'YES' ELSE 'NO' END AS is_tracked_by_cdc - FROM sys.tables - WHERE is_tracked_by_cdc = 1; - - -- View SRTE tables with Change Data Capture enabled - USE NBS_SRTE; - SELECT - name, - CASE WHEN is_tracked_by_cdc = 1 THEN 'YES' ELSE 'NO' END AS is_tracked_by_cdc - FROM sys.tables - WHERE is_tracked_by_cdc = 1; - ``` - - The following images show expected query results for a successful Change Data Capture configuration. - -
-
-
Change Data Capture tables (NBS_ODSE)
- Query results showing 19 Change Data Capture enabled tables in NBS_ODSE, all with is_tracked_by_cdc set to YES -
-
-
Change Data Capture tables (NBS_SRTE)
- Query results showing 44 Change Data Capture enabled tables in NBS_SRTE, all with is_tracked_by_cdc set to YES -
-
- -1. **Back up all databases.** Before going live, take backups of `NBS_ODSE`, `NBS_SRTE`, `RDB`, and `rdb_modern` (if applicable). +1. Apply bootstrap script [101 to enable CDC on NBS_ODSE and NBS_SRTE](https://github.com/CDCgov/NEDSS-DataReporting/blob/v7.13/bootstrap/101-enable_cdc_on_odse_srte_databases-001.sql). ## Deploy RTR services -Now that you have completed database setup and onboarding, deploy the RTR services in the following order. Each service depends on the previous one completing successfully before deployment begins. +Now that you have completed database setup and onboarding, deploy the RTR services in the following order. Some services depend on the previous ones completing successfully before deployment begins. -> Confirm that Kubernetes secrets exist for each RTR service user and the admin user before deploying. If you have not yet created them, see [Create service users and database objects](#create-service-users-and-database-objects). -{: .important } - -1. [Liquibase](../../deploy-nbs7/real-time-reporting/liquibase.html): Skip this step if you used the upgrade script path in [Create required database objects](#create-service-users-and-database-objects). 1. [Debezium](../../deploy-nbs7/real-time-reporting/debezium.html) 1. [Kafka connector](../../deploy-nbs7/real-time-reporting/kafka-connector.html) -1. [Java services](../../deploy-nbs7/real-time-reporting/rtr-java-services.html) - -[nedss-datareporting-liquibase-service]: -[nedss-datareporting-onboarding-user-scripts]: -[nedss-helm-k8-secrets-manifest]: -[nedss-datareporting-manual-deployment]: -[nedss-datareporting-onboarding-data-load]: +1. [Java service](../../deploy-nbs7/real-time-reporting/rtr-java-services.html) --- - -## After onboarding: database upgrades - -Database upgrades apply schema changes required by each NBS 7 release. Run database upgrades when you update NBS 7 to a new release version. - -- If you chose the **Liquibase** path during onboarding, no action is needed. The Liquibase container applies schema changes automatically with each release. -- If you chose the **upgrade scripts** path, navigate to [02_onboarding_script_data_load][nedss-datareporting-onboarding-data-load] and run all of the scripts in the order listed in the repository. Onboarding scripts are excluded from upgrade runs. diff --git a/docs/deploy-nbs7/real-time-reporting/rtr-java-services.md b/docs/deploy-nbs7/real-time-reporting/rtr-java-services.md index 7210b5d..86e128a 100644 --- a/docs/deploy-nbs7/real-time-reporting/rtr-java-services.md +++ b/docs/deploy-nbs7/real-time-reporting/rtr-java-services.md @@ -1,21 +1,20 @@ --- -title: Java services +title: Java service layout: page parent: Deploy real-time reporting nav_order: 4 -description: Covers deployment of RTR Java services that transform Kafka events and load reporting datamarts. +description: Covers deployment of the RTR Java service that transforms Kafka events and loads reporting datamarts. redirect_from: - /docs/7_feature_preview/4_rtr_java_reporting_services.html - /docs/7_feature_preview/4_rtr_java_reporting_services/ --- -# Deploy real-time reporting (RTR) Java services - -> The Java reporting services are being consolidated in an upcoming release. The service validation URLs reflect the NBS 7.12 configuration. -{: .warning } +# Deploy real-time reporting (RTR) Java service This page covers deploying the RTR Java services that process streamed events from Kafka and load domain-specific reporting data. Before proceeding, schedule a maintenance window and notify users that NBS will be unavailable. Database changes made while RTR services are being deployed might not propagate to your reporting database. +Deploying the Java service is a two-phase process. The first deployment seeds the `nrt_*` caching tables that RTR depends on. Once seeding is complete, you reinstall the chart with post-processing enabled. + ## On this page {: .no_toc .text-delta } @@ -25,112 +24,77 @@ This page covers deploying the RTR Java services that process streamed events fr > These steps require a Unix-compatible shell. On Windows, use Git Bash, WSL, or an equivalent terminal emulator. {: .note } -## Installing RTR Java services - -Follow these steps to configure and deploy the RTR Java services Helm chart. - > Verify that you are connected to the correct Kubernetes cluster before proceeding. To confirm, run `kubectl config current-context`. {: .important } -1. Locate the Helm chart for all RTR Java services in the [NEDSS-Helm repository][nedss-helm-rtr-chart]. +## Configure values.yaml -1. Validate the Kubernetes secret for database credentials: +1. Locate the Helm chart for the RTR Java service in the [NEDSS-Helm repository][nedss-helm-rtr-chart]. - ```bash - kubectl get secret/database-access -o yaml - ``` +1. Edit the "EXAMPLE*" placeholder values in `values.yaml` and replace with the correct values for your environment — database connection, Kafka cluster, and any environment-specific overrides. - > Verify that the secret contains the correct database username and password, Kafka cluster, and other required configuration values. If the secret does not exist, create it by applying the provided YAML file. Replace all placeholder values before running: - > - > Script location: [NEDSS-Helm/nbs-secrets.yaml][nedss-helm-k8-secrets-manifest] - > - > ```bash - > kubectl apply -f k8-manifests/nbs-secrets.yaml - > ``` - > - {: .note } +## Initial deployment -1. Validate the image repository: +Install the Helm chart with post-processing disabled: - ```yaml - global.image.repository: "quay.io/us-cdcgov/cdc-nbs-modernization/data-reporting-service" - ``` +```bash +helm install -f reporting-pipeline-service/values.yaml reporting-pipeline-service ./reporting-pipeline-service/ +``` -1. Update feature flags for each service. Verify that `PHCMartETL.bat` is turned off before enabling updates to the PublicHealthCaseFact datamart via RTR: +Verify the pods are running: - ```yaml - featureFlag: - investigation-reporting: - phcDatamartEnable: '''true''' - ``` +```bash +kubectl get deployment reporting-pipeline-service +``` -1. Install the Helm chart for all RTR Java services: +Expected output: - ```bash - helm install rtr . -f values.yaml - ``` +```text +NAME READY UP-TO-DATE AVAILABLE AGE +reporting-pipeline-service 1/1 1 1 16m +``` -1. Verify the pods are running: +## Monitor seeding progress - ```bash - kubectl get pods - ``` +The `/actuator/lag` endpoint reports how far behind the service is in consuming its Kafka topics. Use it to determine when initial seeding is complete. - Expected output: +Retrieve information on reporting-pipeline-service lag in your browser. Replace `` with your actual domain (see [Deploy Traefik ingress controller](../../deploy-nbs7/initial-kubernetes-deployment/initial-kubernetes-deployment.html#deploy-traefik-ingress-controller)): - ```text - NAME READY STATUS RESTARTS AGE - rtr-java-services-investigation-reporting- 1/1 Running 0 2m6s - rtr-java-services-ldfdata-reporting- 1/1 Running 0 2m6s - rtr-java-services-observation-reporting- 1/1 Running 0 2m6s - rtr-java-services-organization-reporting- 1/1 Running 0 2m6s - rtr-java-services-person-reporting- 1/1 Running 0 2m6s - rtr-java-services-post-processing-reporting- 1/1 Running 0 2m6s - ``` +```text + https://data./reporting-pipeline-svc/actuator/lag +``` -1. Validate the services. Replace `` with your actual domain (see [Deploy Traefik ingress controller](../../deploy-nbs7/initial-kubernetes-deployment/initial-kubernetes-deployment.html#deploy-traefik-ingress-controller)): +When all "messagesQueued" values are `0`, seeding is complete. - **investigation-svc** +## Final deployment - ```text - https://data./reporting/investigation-svc/status - Expected: Investigation Service Status OK - ``` +Reinstall the chart with post-processing enabled: - **person-svc** +1. Update `values.yaml` to enable post-processing: - ```text - https://data./reporting/person-svc/status - Expected: Person Service Status OK + ```yaml + featureFlag: + postProcessingEnable: "true" ``` - **observation-svc** +1. Upgrade the release: - ```text - https://data./reporting/observation-svc/status - Expected: Observation Service Status OK + ```bash + helm upgrade -f reporting-pipeline-service/values.yaml reporting-pipeline-service ./reporting-pipeline-service/ ``` - **organization-svc** - ```text - https://data./reporting/organization-svc/status - Expected: Organization Service Status OK - ``` +1. Verify the pods restarted cleanly: - **ldfdata-svc** - - ```text - https://data./reporting/ldfdata-svc/status - Expected: LDFData Service Status OK + ```bash + kubectl rollout status deployment/reporting-pipeline-service + kubectl get deployment reporting-pipeline-service ``` - **post-processing-svc** +1. Confirm the service is healthy. Replace `` with your actual domain (see [Deploy Traefik ingress controller](../../deploy-nbs7/initial-kubernetes-deployment/initial-kubernetes-deployment.html#deploy-traefik-ingress-controller)): ```text - https://data./reporting/post-processing-svc/status - Expected: Post Processing Service Status OK + https://data./reporting-pipeline-svc/actuator/health ``` [nedss-helm-rtr-chart]: -[nedss-helm-k8-secrets-manifest]: