Watchlists maintainer, sync all watchlists on setup by CAWilson94 · Pull Request #260144 · elastic/kibana

CAWilson94 · 2026-03-28T15:06:51Z

Summary

This PR adds the watchlist maintainer and registers it on plugin setup. Additionally, all watchlists are synced on a 10 minute schedule.

Feature flags:

xpack.securitySolution.enableExperimental:
  - entityAnalyticsEntityStoreV2
  - entityAnalyticsWatchlistEnabled

Debugging log config:

logging:
  loggers:
    - name: plugins.securitySolution
      level: debug
    - name: plugins.entityStore
      level: debug

Document generator commands:

yarn start organization-quick
yarn start generate-entity-maintainers-data --quick

Example watchlist with entity source

// watchlist 
POST kbn:/api/entity_analytics/watchlists 
{
    "name": "test-name-1",
    "description": "Test watchlist for index sync-1",
    "riskModifier": 1.5,
    "managed": false
}
// get the id for the next call
GET kbn:/api/entity_analytics/watchlists/list

// entity source for this watchlist 
POST kbn:/api/entity_analytics/watchlists/{watchlist-id}/entity_source
{
    "type": "index",
    "name": "some-watchlist-source-name",
    "queryRule": "entity.risk.calculated_level : \"High\" or entity.risk.calculated_level : \"Moderate\"",
    "identifierField": "user.name"
}


GET .entity-analytics.watchlists.test-name-1-default/_search

Testing Steps:
0. Update WATCHLIST_MAINTAINER_INTERVAL in register watchlist maintainer to something smol like 1m

Start up kibana with the admin/platinum license.
Ensure entity analytics is enabled (v2)
With the data generated from document generator ...
Look in kibana logs for Entity Analytics Watchlist Maintainer
Head over to dev tools and create a watchlist with an entity source
Head back to the kibana logs, look out for:
'Watchlist maintainer run for namespace "default'
Then something like
'Found n watchlist(s) to sync in namespace "default"
Syncing watchlist "name" ()'
Go back to dev tools and have a look at the watchlist index to see synced entities!

CAWilson94 · 2026-03-30T14:40:50Z

+    logger.debug(`Initializing watchlist maintainer for namespace "${namespace}"`);
+
+    // TODO: add watchlist-specific setup logic here (e.g. saved objects, data client init)
+    // Add in call to pre-built watchlists install


Doing this in the entity analytics migration flow instead

jaredburgettelastic

👍 Code looks good, but I think we will have to register this as a task in check_registered_task_types.ts. Will wait for CI to pass/fail to check first, as I'm unsure if that's necessary or not until the feature flag is enabled by default

elasticmachine · 2026-03-30T20:26:42Z

💔 Build Failed

Buildkite Build
Commit: 14cbaeb

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #63 / Cloud Security Posture GET /internal/cloud_security_posture/stats KSPM Compliance Dashboard Stats API should return KSPM benchmarks V2
[job] [logs] FTR Configs #98 / serverless security UI Security ML Trained models list page navigation renders trained models list
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.eqlRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.eqlRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.eqlRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.indicatorRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.indicatorRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.indicatorRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.mlRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.mlRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.mlRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.newTermsRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.newTermsRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.newTermsRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.queryRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.queryRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.queryRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.savedQueryRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.savedQueryRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.savedQueryRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.thresholdRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.thresholdRule
[job] [logs] Jest Integration Tests #1 / Serverless upgrade and rollback checks detect param changes to review for: siem.thresholdRule

Metrics [docs]

✅ unchanged

History

cc @CAWilson94

tiansivive · 2026-03-30T20:28:17Z

+    // credentials of the user who installed the Entity Store) rather than
+    // buildScopedInternalSavedObjectsClientUnsafe, which builds its own fake
+    // request with empty headers and no auth credentials.
+    const soClient = coreStart.savedObjects.getScopedClient(fakeRequest, {


I think we now have a better way of doing this. Maybe even privmon or risk engine already use the new version, so we can check later

Just flagging it, wont block for this!

Thanks, I think you are correct . Will add it to the list!

Agreed, you can use this as an example of the new way to do so for entity maintainers. It comes with its own esClient, scoped to the user who originally enabled the entity store.

(we now get this for free from task manager)

CAWilson94 · 2026-03-30T20:33:53Z

👍 Code looks good, but I think we will have to register this as a task in check_registered_task_types.ts. Will wait for CI to pass/fail to check first, as I'm unsure if that's necessary or not until the feature flag is enabled by default

Thanks @jaredburgettelastic nice catch!

The check_registered_task_types.ts test only sees task types that are registered with default config.

Since both entityAnalyticsEntityStoreV2 and entityAnalyticsWatchlistEnabled default to false, the watchlist maintainer task type (entity_store:v2:entity_maintainer_task:watchlist) won't be registered when that test runs — so we don't need to add it there yet.

Same for the risk scoring maintainer, so we will need to update both once the flags are gone.

### Summary This PR adds the watchlist maintainer and registers it on plugin setup. Additionally, all watchlists are synced on a 10 minute schedule. **Feature flags:** ``` xpack.securitySolution.enableExperimental: - entityAnalyticsEntityStoreV2 - entityAnalyticsWatchlistEnabled ``` Debugging log config: ``` logging: loggers: - name: plugins.securitySolution level: debug - name: plugins.entityStore level: debug ``` Document generator commands: ``` yarn start organization-quick yarn start generate-entity-maintainers-data --quick ``` Example watchlist with entity source ``` // watchlist POST kbn:/api/entity_analytics/watchlists { "name": "test-name-1", "description": "Test watchlist for index sync-1", "riskModifier": 1.5, "managed": false } // get the id for the next call GET kbn:/api/entity_analytics/watchlists/list // entity source for this watchlist POST kbn:/api/entity_analytics/watchlists/{watchlist-id}/entity_source { "type": "index", "name": "some-watchlist-source-name", "queryRule": "entity.risk.calculated_level : \"High\" or entity.risk.calculated_level : \"Moderate\"", "identifierField": "user.name" } GET .entity-analytics.watchlists.test-name-1-default/_search ``` Testing Steps: 0. Update WATCHLIST_MAINTAINER_INTERVAL in register watchlist maintainer to something smol like 1m 1. Start up kibana with the admin/platinum license. 2. Ensure entity analytics is enabled (v2) 3. With the data generated from document generator ... 4. Look in kibana logs for Entity Analytics Watchlist Maintainer 5. Head over to dev tools and create a watchlist with an entity source 6. Head back to the kibana logs, look out for: 'Watchlist maintainer run for namespace "default' Then something like 'Found n watchlist(s) to sync in namespace "default" Syncing watchlist "name" (<watchlist-id>)' 7. Go back to dev tools and have a look at the watchlist index to see synced entities! --------- Co-authored-by: Jared Burgett <147995946+jaredburgettelastic@users.noreply.github.com>

maintainer WiP

20b15c3

CAWilson94 mentioned this pull request Mar 28, 2026

[Entity Analytics] [Watchlists] Install prebuilt watchlists on setup #259515

Closed

CAWilson94 changed the title ~~maintainer WiP~~ Watchlists Maintainer and Prebuilt Watchlist Install Mar 30, 2026

Merge branch 'main' into watchlist-maintainer

6883a56

CAWilson94 commented Mar 30, 2026

View reviewed changes

CAWilson94 added 2 commits March 30, 2026 20:04

Watchlists maintainer with sync all watchlists

e04e6db

entity source sync all test

0cac813

CAWilson94 changed the title ~~Watchlists Maintainer and Prebuilt Watchlist Install~~ Watchlists maintainer, sync all watchlists on setup Mar 30, 2026

CAWilson94 marked this pull request as ready for review March 30, 2026 19:08

CAWilson94 requested review from a team as code owners March 30, 2026 19:08

CAWilson94 requested a review from hop-dev March 30, 2026 19:08

CAWilson94 self-assigned this Mar 30, 2026

Merge branch 'main' into watchlist-maintainer

14cbaeb

jaredburgettelastic approved these changes Mar 30, 2026

View reviewed changes

tiansivive reviewed Mar 30, 2026

View reviewed changes

tiansivive approved these changes Mar 30, 2026

View reviewed changes

Merge branch 'main' into watchlist-maintainer

7f129fb

jaredburgettelastic merged commit ad8803c into main Mar 30, 2026
18 checks passed

jaredburgettelastic deleted the watchlist-maintainer branch March 30, 2026 22:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Watchlists maintainer, sync all watchlists on setup #260144

Watchlists maintainer, sync all watchlists on setup #260144
jaredburgettelastic merged 6 commits intomainfrom
watchlist-maintainer

CAWilson94 commented Mar 28, 2026 •

edited

Loading

Uh oh!

CAWilson94 Mar 30, 2026

Uh oh!

jaredburgettelastic left a comment •

edited

Loading

Uh oh!

elasticmachine commented Mar 30, 2026 •

edited

Loading

Uh oh!

tiansivive Mar 30, 2026

Uh oh!

CAWilson94 Mar 30, 2026

Uh oh!

jaredburgettelastic Mar 30, 2026

Uh oh!

jaredburgettelastic Mar 30, 2026

Uh oh!

CAWilson94 commented Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

CAWilson94 commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

CAWilson94 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

jaredburgettelastic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

tiansivive Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

CAWilson94 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

jaredburgettelastic Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

jaredburgettelastic Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

CAWilson94 commented Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CAWilson94 commented Mar 28, 2026 •

edited

Loading

jaredburgettelastic left a comment •

edited

Loading

elasticmachine commented Mar 30, 2026 •

edited

Loading