[doc](fe) Add data lineage documentation by seawinde · Pull Request #3997 · apache/doris-website

seawinde · 2026-07-17T09:51:50Z

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:
The website does not provide a complete technical reference for Apache Doris
data lineage or an end-to-end guide that developers can follow to implement,
package, deploy, and verify a lineage plugin.

Change Summary:

Add English and Chinese data lineage technical guides for dev and 4.x,
covering the collection flow, event semantics, supported DML, FE
configuration, queue behavior, runnable SQL examples, and troubleshooting.
Add an English and Chinese source-tree tutorial for a logging lineage plugin,
including Maven layout, Java implementations, service registration,
packaging, deployment, verification, rollback, and external governance
integration options such as OpenMetadata and OpenLineage.
Add localized architecture and event-model diagrams.
Add dev, 4.x, Community, and Chinese Community navigation entries.

Design rationale:
The SQL manual explains the user-facing behavior and lineage model, while the
Community guide provides the implementation workflow. Dev and 4.x pages remain
aligned except for version-specific availability wording, and executable code
blocks are identical between English and Chinese pages.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason

Manual validation:

Executed the documented SQL example and verified its result.
Compiled the Java snippets against Apache Doris 4.0.6, built the plugin JAR,
and verified that the directory plugin loader loaded and executed it.
Validated all Shell snippets with bash -n and parsed all XML blocks.
Verified English and Chinese routes, images, and page layout locally.
Ran changed-scope link, SEO, i18n, feature, and whitespace checks.
The latest master full English build is currently blocked by malformed
generated release metadata already present outside this change; the build
reports no errors in the data lineage files.
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. This PR adds the documentation.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: The website did not provide a complete reference for Doris data lineage or an end-to-end guide for developing a lineage plugin. Change Summary: - Add English and Chinese data lineage technical guides for dev and 4.x, including event semantics, configuration, examples, operations, and troubleshooting. - Add an English and Chinese source-tree tutorial for a logging lineage plugin and external governance integration. - Add localized architecture diagrams and navigation entries. ### Release note None ### Check List (For Author) - Test: Manual test - Validated SQL examples, Java compilation and plugin loading. - Validated shell and XML snippets, page routes, and images. - Ran changed-scope links, SEO, i18n, and feature checks. - Behavior changed: No. - Does this need documentation: Yes. This is the documentation PR.

seawinde · 2026-07-17T09:52:04Z

run buildall

morrySnow · 2026-07-17T10:43:24Z

+| `INSERT OVERWRITE ... SELECT` | Overwrite 成功后生成一个事件。 |
+| `CREATE TABLE AS SELECT` | 内部 Insert 成功后生成一个事件。 |
+
+以下操作不会生成事件：`SELECT`、`UPDATE`、`DELETE`、导入任务、仅包含 `VALUES` 的 Insert，以及目标表为 `__internal_schema` 的写入。


UPDATE、DELETE 也不会吗？特意排除了？

morrySnow · 2026-07-17T10:48:51Z

@@ -0,0 +1,336 @@
+---
+{
+    "title": "数据血缘原理与使用",


这个为什么在sql操作手册下面呢？可能放在 docs/data-governance 下比较合适？

morrySnow

Thanks for this comprehensive data lineage documentation! I've cross-referenced it against the actual code implementation at fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/. The documentation is largely accurate, but I found a few issues that should be addressed:

Summary of Findings

JAR name inconsistency between user guide and developer guide (see inline comment).
INSERT OVERWRITE syntax is missing the TABLE keyword throughout (see inline comments).
PluginContext property access description is imprecise (see inline comment).
"FE trims surrounding spaces" claim for activate_lineage_plugin parsing should be verified or removed.
"Every active plugin" phrasing is slightly misleading — the worker dispatches to every loaded plugin, and eventFilter() decides per-plugin processing.

Verified as Correct ✅

LineagePlugin / LineagePluginFactory interface signatures match the code exactly
LineageInfo data structure (direct, indirect, dataset lineage maps) matches code
LineageContext fields (sourceCommand, queryId, user, clientIp, database, catalog, state, timestampMs, durationMs, externalCatalogProperties) all match
eventFilter() is indeed called twice — once in query thread (via hasActivePlugins() before extraction) and once in worker thread (before exec())
Queue size default of 50000 (Config.lineage_event_queue_size) matches code
Best-effort delivery semantics (no rollback, no retry, no persistence) match code
UPDATE, DELETE, SELECT, load jobs do not generate lineage events — confirmed by checking InsertIntoTableCommand, InsertOverwriteTableCommand, CreateTableCommand only
VALUES-only inserts and __internal_schema target tables are skipped — confirmed in LineageUtils.shouldSkipLineage()
Worker uses eventQueue.poll(5, SECONDS) with 5-second timeout — matches description
Empty activate_lineage_plugin instantiates ALL discovered factories — confirmed in discoverPlugins() logic
Plugin discovery only at FE startup, no dynamic reload — confirmed
Single daemon worker thread — confirmed (new Thread(new Worker(), "LineageEventProcessor"))
Log keywords in troubleshooting table all match the actual code
CTE resolution and UNION branch expansion are performed — confirmed in LineageInfoExtractor
Plugin extends AutoCloseable with close() default method — matches code
LineagePluginFactory.create(context) defaults to delegating to create() — matches code

morrySnow · 2026-07-17T11:01:26Z

+The following operations do not generate an event: `SELECT`, `UPDATE`, `DELETE`, load jobs, `VALUES`-only inserts, and writes whose target is `__internal_schema`.
+
+:::caution Delivery guarantee
+


The SQL syntax used throughout this document (INSERT OVERWRITE ... SELECT) drops the TABLE keyword. The standard Doris SQL syntax is INSERT OVERWRITE TABLE table_name ... SELECT .... The command class in the code is InsertOverwriteTableCommand. While this may be intended as shorthand in prose, the supported-statements table should use the precise syntax. The SVG diagrams (lineage-architecture.svg, lineage-event-model.svg) also use INSERT OVERWRITE SELECT without TABLE.

morrySnow · 2026-07-17T11:01:26Z

+```
+
+Configure `fe.conf` on every FE. Replace `example-lineage` with the value returned by the plugin factory's `name()` method:
+


The JAR filename example-lineage-plugin.jar is inconsistent with the developer guide (community/developer-guide/data-lineage-plugin-development.md), where the Maven <finalName> produces example-lineage.jar. Users who follow both guides may be confused about which JAR name is correct. Please align the JAR name across both documents — either change this to example-lineage.jar or update the dev guide's pom.xml <finalName> and build output references.

morrySnow · 2026-07-17T11:01:26Z

+activate_lineage_plugin = example-lineage,governance-lineage
+```
+
+At startup, FE discovers built-in and external factories, then creates instances only for the configured names. Each FE reads its own configuration, so every FE that can execute DML must use the same plugin set. Restart FE after a change. This setting cannot be changed dynamically through `ADMIN SET FRONTEND CONFIG`.


This paragraph says: "Separate multiple plugin names with commas; FE trims surrounding spaces." Looking at the code, Config.activate_lineage_plugin is a String[], and discoverPlugins() uses Arrays.asList(Config.activate_lineage_plugin) followed by new HashSet<>(...). The raw Arrays.asList() does NOT trim whitespace from individual elements. Unless the Doris Config framework's String[] parser does the trimming (please verify), this claim may be incorrect. If a user writes activate_lineage_plugin = example-lineage, governance-lineage, the second entry would be " governance-lineage" (with leading space) and would fail name matching because the comparison is case-sensitive HashSet.contains().

morrySnow · 2026-07-17T11:01:26Z

+
+`LineageInfo` contains a target table, target output columns, source tables, direct lineage, indirect lineage, and a `LineageContext`.
+
+| Part | API | Meaning |


The phrase "PluginContext contains plugin.name and plugin.path" is imprecise. Looking at the PluginContext class (fe-extension-spi/.../PluginContext.java), it only exposes getProperties() which returns a Map<String, String>. The properties "plugin.name" and "plugin.path" are stored as map entries (set in discoverPlugins() via props.put("plugin.path", ...) and props.put("plugin.name", ...)). A developer reading this would look for context.getPluginName() or context.pluginName which don't exist. Please clarify: accessed via context.getProperties().get("plugin.name") and context.getProperties().get("plugin.path"). Same issue exists in the Chinese version at the corresponding position.

morrySnow · 2026-07-17T11:01:26Z

+
+`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`.
+
+At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through `plugin.path`, and initialize its resources there.


Same INSERT OVERWRITE issue as the user guide — the TABLE keyword is missing. Standard Doris syntax is INSERT OVERWRITE TABLE ... SELECT .... The command class name InsertOverwriteTableCommand also confirms this. Please use the precise syntax throughout (applies to all files: EN user guide, ZH user guide, ZH dev guide, and all versioned copies).

morrySnow · 2026-07-17T11:01:26Z

+  <text x="96" y="226" class="label">Supported DML</text>
+  <text x="96" y="262" class="body">INSERT INTO SELECT</text>
+  <text x="96" y="290" class="body">INSERT OVERWRITE SELECT</text>
+  <text x="96" y="318" class="body">CREATE TABLE AS SELECT</text>


The SVG text uses INSERT OVERWRITE SELECT without the TABLE keyword. The standard Doris SQL syntax is INSERT OVERWRITE TABLE ... SELECT .... Please add TABLE to match the actual syntax. Affects both lineage-architecture.svg and its zh-CN variant.

Move the user guide under the top-level data governance section. Align INSERT OVERWRITE syntax and example JAR names across guides and diagrams. Clarify PluginContext property access, plugin filtering, configuration parsing, and UPDATE/DELETE event behavior. Synchronize the related dev and 4.x Key Features descriptions.

seawinde temporarily deployed to Production July 17, 2026 09:51 — with GitHub Actions Inactive

morrySnow reviewed Jul 17, 2026

View reviewed changes

seawinde temporarily deployed to Production July 19, 2026 06:13 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[doc](fe) Add data lineage documentation#3997

[doc](fe) Add data lineage documentation#3997
seawinde wants to merge 2 commits into
apache:masterfrom
seawinde:docs/data-lineage

seawinde commented Jul 17, 2026

Uh oh!

seawinde commented Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow left a comment

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		The following operations do not generate an event: `SELECT`, `UPDATE`, `DELETE`, load jobs, `VALUES`-only inserts, and writes whose target is `__internal_schema`.

		:::caution Delivery guarantee

		```

		Configure `fe.conf` on every FE. Replace `example-lineage` with the value returned by the plugin factory's `name()` method:


		`LineageInfo` contains a target table, target output columns, source tables, direct lineage, indirect lineage, and a `LineageContext`.

		\| Part \| API \| Meaning \|


		`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`.

		At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through `plugin.path`, and initialize its resources there.

Uh oh!

Conversation

seawinde commented Jul 17, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

seawinde commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morrySnow left a comment

Choose a reason for hiding this comment

Summary of Findings

Verified as Correct ✅

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants