#1142 Use Arranger SearchClient by demariadaniel · Pull Request #1145 · nci-hcmi-catalog/portal

demariadaniel · 2026-03-09T19:38:00Z

Updates HCMI application to run using Arranger SearchClient with OpenSearch

Replaces use of elastic-search client with Arranger Server SearchClient
Updates Mongoose client
Removes Mongoosastic library and uses explicit client indexing
Fixes bugs & cleanup related to legacy code
Updates Lodash patch version
Adds explicit handling for 'last-updated' index

This reverts commit 99cfd46.

…into 1142/feat-support-opensearch

demariadaniel · 2026-04-07T20:11:12Z

 const growthChartPalette = ['#ef5350', '#80deea', '#3949ab'];

-const multipleModelsChartPalette = ['#d4e157', '#0277bd'];
+const multipleModelsChartPalette = ['#d4e157', '#0277bd', '#f06292'];


Added a 3rd value for differentiating 'No Data' from 'No'

mistryrn

Still need to complete my local testing once the docker-compose change we discussed on Slack is implemented (adding OpenSearch), but I've reviewed the code changes and they look great! Just a few minor change suggestions and cleanup/documentation asks. Other than this (and pending local testing), I think we're good to go 🙂

Awesome job, Dan! 🎉

mistryrn · 2026-04-08T16:19:20Z

+    if (value && typeof value === 'object') {
+      if (Array.isArray(value)) {
+        const firstEntry = value[0];
+        const cleanedVals =
+          firstEntry && typeof firstEntry === 'object'
+            ? value.map((val) => {
+                const cleanedValue = _.omit(val, mongoKeys);
+                return cleanedValue;
+              })
+            : value;
+        cleanedDoc[key] = cleanedVals;
+      } else if (value instanceof mongoose.Types.ObjectId) {
+        cleanedDoc[key] = value.toString();
+      } else if (!(value instanceof Date)) {
+        const cleanedValue = _.omit(value, mongoKeys);
+        cleanedDoc[key] = cleanedValue;
+      }
+    }


Thanks for working all this logic out, it must have been quite a bit of trial and error!

For future developers' reference, would you mind updating the TSDoc comment above this function to describe why we're cleaning/handling objects, arrays, mongoose ObjectIds, and Dates differently? Either that, or put some inline comments above each conditional explaining the purpose/logic, or both!

This is definitely the kind of thing that, if documented well, will be greatly appreciated by whoever looks at this months from now 🙂

Comments added, let me know if you would revise these

mistryrn · 2026-04-08T16:30:41Z

-import variantsIndexConfig from '../../elasticsearch/variantsIndex.json' with { type: "json" };
-const modelsIndexName = process.env.ES_INDEX || pm2.ES_INDEX || 'hcmi';
 const esHost = process.env.ES_HOST || `${pm2.ES_HOST}:${pm2.ES_PORT}`;
+const updateIndexName = process.env.ES_UPDATE_INDEX || pm2.ES_UPDATE_INDEX || 'hcmi-update';


Nice catch explicitly handling the last updated index 👍

mistryrn

Couple small suggestions for the docker-compose file 🙂

mistryrn

Found one bug with the republish script in my local testing: looks like one of the deleted ES schema files in /cms was used there. Hopefully it'll be a quick fix to get that working again 🙏

Also suggested a minor logs cleanup.

mistryrn · 2026-04-09T13:45:37Z

Related to my previous comment: looks like the republish script relied on this export as a dependency. We may need to restore it or use an alternative to get that working again.

mistryrn

Thanks for addressing the feedback so far! I encountered a few more issues in my local testing today:

(1) The "Updated" column on the /admin CMS table and the "Updated" timestamp shown on the edit model view don't seem to be getting updated any more when saving or publishing a model. This isn't a blocker for testing on the Dev server, but is something we'll need to fix before we go to Prod.

(2) I'm encountering some data/index-related issues when loading the public-facing UI pages. These might be a blocker for testing on the Dev server, as we might encounter these when we go to test. I know you've fixed similar issues on your local already, so I'm going to leave some instructions that might help reproduce them/recreate the scenario of setting up a new OpenSearch instance.
- a. The "Search By Model Name" filter on the Search page is failing with 400 errors on the ModelQuickSearch query.
  - code: "GRAPHQL_VALIDATION_FAILED"
  - message: 'Cannot query field "autocomplete" on type "modelNode".'
- b. The "Model Details" page fails to load when I click on a model from the Search page table. I'm getting 400 errors for ModelDataQuery and VariantsData:
  - ModelDataQuery errors:
    - message: 'Cannot query field "distributor_part_number" on type "modelNode".'
    - message: 'Cannot query field "proteomics_url" on type "modelNode".'
    - message: 'Cannot query field "hits" on type "modelFiles".'
    - message: 'Cannot query field "matched_models" on type "modelNode". Did you mean "has_matched_models" or "matched_models_list"?'
    - code: "GRAPHQL_VALIDATION_FAILED" (same code for each error)
  - VariantsData errors:
    - message: 'Cannot query field "variants" on type "modelNode".'
    - code: "GRAPHQL_VALIDATION_FAILED"
(3) The publish validation seems to be preventing models that were previously valid from getting published. The following 2 fields are no longer allowing null values, despite the Prod Catalog containing many published examples of null or N/A for them:
- growth_rate
- age_at_diagnosis
To get around (3) locally, I added .nullable(true) to both fields under getPublishSchema.

I know I've seen the issues under (2) addressed on your local, so maybe it's something you'll only see if you're setting things up from scratch in OpenSearch or when attempting to migrate data from Elasticsearch?

To reproduce the issues under (2), the quickest way is to run the repairEs script, as this will delete your current indices, initialize new ones, and then republish all published models in your db.

# from project root
ENV=prd npm run repairEs

This should simulate starting from a fresh OpenSearch instance/empty indices and should hopefully reproduce the issues in (2).

Note that you'll likely need to trigger a variant/gene index publish after this to get the "Search by Genes" filters working on the home page. Do this by editing a model in the CMS, opening the "Variants" tab and use the "+ Research Somatic Variants" dropdown to do an "Automatic Import from GDC"

If you are able to reproduce the issues under (2), I suspect we might need to revisit some of the new model/publish schemas. Comparing things against the 1.26.2 tag might be a good way to double-check that we're including everything we need: https://github.com/nci-hcmi-catalog/portal/tree/1.26.2

One thing I noticed when comparing an older ES index to a new OS index: looks like there were 3 fields related to matched models before.

matched_models (array)
has_matched_models (boolean)
matched_models_list (string list of relevant matched models, containing the name of the current model)

I'm not sure if that might be related to the errors I'm getting loading the Model Details page, but it might be worth looking into.

mistryrn

Thanks for addressing the "Last Updated" and "publish validation" issues from my previous feedback. I've re-tested locally and can confirm that those are mostly resolved 🎉

Additionally, the index issues I encountered during my testing last week are no longer occurring! Looking back, it may have been related to my running both ES and OS locally at the same time on Thursday 🙈

Just two more small issues, but these won't block testing on the new dev server:

(1) "Last Updated" is still not getting updated on Save. Thanks for fixing it on Publish, but we should also be updating this on Save as well for consistency with the existing behaviour (and to make it easier to find the last model worked on if it wasn't published right away).
(2) Individual model "Delete" functionality is currently broken (bulk delete still works). This is unrelated to any of your code changes, but instead a consequence of the Mongo/Mongoose 8 upgrade. Looks like Mongoose 8 relies on a dependency mquery@5.0.0 that tries to use a deprecated method .findOneAndRemove (which, consequently, was removed in Mongoose 8) 🙃
- Not a huge deal, the fix would just be to upgrade to Mongoose 9, which uses mquery@6.0.0, which no longer references the deprecated methods.
- I skimmed the breaking changes for Mongoose 8 → 9, the main change that might impact us is "Update pipelines disallowed by default": https://mongoosejs.com/docs/migrating_to_9.html#update-pipelines-disallowed-by-default
  - We use this functionality in a few of our migrations, but we should be able to just update them by adding , { updatePipeline: true }
- If upgrading to Mongoose 9 breaks too many things, we could alternatively use a module resolution in the root package.json to force mquery to resolve to 6.0.0

Let's try to get (1) resolved ASAP, but (2) can be spun out into a separate ticket and we can inform the NIH of the temporary workaround if we don't resolve it before going to Prod.

Just leaving some reference for (2) below:

This reverts commit f017073.

Testing Feb 6

3fd77ec

demariadaniel self-assigned this Mar 9, 2026

demariadaniel marked this pull request as draft March 9, 2026 19:38

demariadaniel added 12 commits March 9, 2026 17:10

Import organize

f7cd504

Use Mongoose 8.23

3ea8145

Updated searchClient names, imports, & usage -- draft

ed200f1

Fixed configureSets

650738e

mongoosastic, publish, and promise updates

f53c403

Genes searchClient & async updates

05e748e

Use module imports pt 1

99cfd46

Revert "Use module imports pt 1"

bdb209a

This reverts commit 99cfd46.

Replace use of logger.audit

18ba7a1

Quick clean up

115deae

First Working Publish

0569dcb

Fixed Unpublish

92e2ae3

demariadaniel commented Mar 23, 2026

View reviewed changes

Comment thread cms/src/services/searchClient/publish.js Outdated

demariadaniel added 14 commits March 23, 2026 17:09

Remove mongoosastic

0ed00b0

Fix lastUpdated index setup issue

72f6171

Fix cms client args

fb35762

Fixed SelectAll Admin Table behaviour

688cace

File import and organization changes

9e00f8e

Fixed bulk publish/unpublish toggle all & counts

cbffa77

Update searchClient definition

a637dc8

Fix duplicate indexing bug

4bf3c8e

Fix Somatic Variant Search

129c691

Merge branch 'develop' of https://github.com/nci-hcmi-catalog/portal …

1374d35

…into 1142/feat-support-opensearch

Genomic Variant falsy handling

fed215b

Fixed Matched Models Publishing

d0b2bac

Merge branch 'develop' of https://github.com/nci-hcmi-catalog/portal …

a0f784e

…into 1142/feat-support-opensearch

MatchedModels UI improvements

d47660c

Use Arranger Server 3.0.3-beta.1, updated comments

1e07988

demariadaniel requested a review from mistryrn April 7, 2026 18:19

demariadaniel marked this pull request as ready for review April 7, 2026 18:19

demariadaniel changed the title ~~WIP: #1142 Use Arranger SearchClient~~ #1142 Use Arranger SearchClient Apr 7, 2026

Revert testing port change

50a27bd

demariadaniel commented Apr 7, 2026

View reviewed changes

mistryrn reviewed Apr 8, 2026

View reviewed changes

demariadaniel added 3 commits April 8, 2026 15:15

Update docker-compose to use Opensearch

ea2d64c

Use kebab case CMS folder name

a5f5824

Remove pnpm workspace, change function name casing

dae22dd

mistryrn reviewed Apr 8, 2026

View reviewed changes

Comment thread docker-compose.yml Outdated

Comment thread docker-compose.yml Outdated

demariadaniel added 2 commits April 8, 2026 16:47

Docker compose edits

a41e205

Sync data formatting

ec859b0

mistryrn requested changes Apr 9, 2026

View reviewed changes

demariadaniel added 3 commits April 9, 2026 10:08

Remove extra logging

e428b3c

Fix republish import & add default index values

593ba23

Additional comments for cleanMongoDoc

3064492

mistryrn reviewed Apr 9, 2026

View reviewed changes

Comment thread cms/src/services/search-client/publish.js Outdated

demariadaniel added 3 commits April 10, 2026 15:35

Fix UpdatedAt field

2955c66

Fix Typo

1d24893

Allow Nullable Fields

283f690

mistryrn reviewed Apr 15, 2026

View reviewed changes

demariadaniel added 8 commits April 16, 2026 11:52

Current UpdateAt Model on Save

87f8632

Use migrations config.js

06fb5f8

Matched Models List value correction

f017073

Revert "Matched Models List value correction"

b3b0dfb

This reverts commit f017073.

Matched Models List bug

e9e5553

Remove logging

65d8d14

Minor doc updates

4db4a4b

NODE_ENV in .schema, revert matched_models change

85e98d3

Conversation

demariadaniel commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

demariadaniel Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

mistryrn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mistryrn Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

demariadaniel Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mistryrn Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mistryrn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mistryrn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mistryrn Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

mistryrn left a comment • edited by demariadaniel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mistryrn left a comment • edited by demariadaniel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

demariadaniel commented Mar 9, 2026 •

edited

Loading

demariadaniel Apr 9, 2026 •

edited

Loading

mistryrn left a comment •

edited by demariadaniel

Loading

mistryrn left a comment •

edited by demariadaniel

Loading