From 73a1f8d4b66d5b141fb352ccd6f8ca9db13c1e26 Mon Sep 17 00:00:00 2001 From: Refrain Date: Thu, 16 Jul 2026 14:53:44 +0800 Subject: [PATCH 1/3] docs: document ALTER ROUTINE LOAD property changes --- .../load-and-export/ALTER-ROUTINE-LOAD.md | 317 +++++++++++++---- .../load-and-export/ALTER-ROUTINE-LOAD.md | 318 +++++++++++++----- 2 files changed, 486 insertions(+), 149 deletions(-) diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md index 007a3ce4821bb..10f683b343e65 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md @@ -2,104 +2,273 @@ { "title": "ALTER ROUTINE LOAD", "language": "en", - "description": "This syntax is used to modify an existing routine load job. Only jobs in PAUSED state can be modified." + "description": "Modifies an existing Routine Load job. Kafka jobs can also switch their target table." } --- ## Description -This syntax is used to modify an existing routine load job. Only jobs in PAUSED state can be modified. +`ALTER ROUTINE LOAD` modifies an existing Routine Load job. For a single-table Kafka job, one statement can switch the target table and modify supported job and Kafka data source properties at the same time. Kinesis jobs support job and data source property modifications, but do not support target-table switching. + +Only a job in the `PAUSED` state can be modified. The recommended sequence is `PAUSE ROUTINE LOAD`, `ALTER ROUTINE LOAD`, and then `RESUME ROUTINE LOAD`. ## Syntax ```sql -ALTER ROUTINE LOAD FOR [.] -[] -FROM [] -[] +ALTER ROUTINE LOAD FOR [.] +[ + SET TARGET TABLE = "" + | [, ...] +] +[PROPERTIES ( + "" = "" + [, ...] +)] +[FROM ( + "" = "" + [, ...] +)]; ``` +The statement must specify at least one modification. + +When `SET TARGET TABLE` is specified, the existing job must be a Kafka job and an optional `FROM` clause must be `FROM KAFKA`. A Kinesis job can use the other ALTER forms but cannot specify `SET TARGET TABLE`. + ## Required Parameters -**1. `[.]`** +### `[.]` + +Specifies the Routine Load job to modify. -> Specifies the name of the job to be modified. The identifier must begin with a letter character and cannot contain spaces or special characters unless the entire identifier string is enclosed in backticks. -> -> The identifier cannot use reserved keywords. For more details, please refer to identifier requirements and reserved keywords. +- If `` is omitted, Doris uses the current database. +- The job name can be enclosed in backticks. ## Optional Parameters -**1. ``** - -> Specifies the job parameters to be modified. Currently supported parameters include: -> -> - desired_concurrent_number -> - max_error_number -> - max_batch_interval -> - max_batch_rows -> - max_batch_size -> - jsonpaths -> - json_root -> - strip_outer_array -> - strict_mode -> - timezone -> - num_as_string -> - fuzzy_parse -> - partial_columns -> - max_filter_ratio - -**2. ``** - -> Properties related to the data source. Currently supports: -> -> - `` -> - `` -> - `` -> - `` -> - Custom properties, such as `` - -**3. ``** - -> The type of data source. Currently supports: -> -> - KAFKA +### `SET TARGET TABLE` + +Switches the target table of a single-table Kafka Routine Load job to ``. + +The target table name must be a string literal, for example, `SET TARGET TABLE = "new_table"`. The old `ON new_table` form and an unquoted `SET TARGET TABLE = new_table` are not supported. + +The target table must meet all of the following requirements: + +- It is in the same database as the Routine Load job. `` cannot specify another database. +- It is a non-temporary OLAP table. +- The job uses Kafka as its data source. Kinesis jobs do not support target-table switching. +- The job is a single-table Routine Load job. Multi-table jobs are not supported. +- The current user has the `LOAD_PRIV` privilege on both the old and new target tables. +- The new table is compatible with the existing column mappings, filters, fixed partitions, merge or delete conditions, sequence column, and partial-update mode. Doris replans the existing load configuration against the new table and rejects the change if a valid load plan cannot be generated. +- If the job was created with `load_to_single_tablet=true`, the new table uses Random Distribution. + +`SET TARGET TABLE` can be combined with `PROPERTIES` and `FROM KAFKA`, but it cannot be combined with `FROM KINESIS` or ``. + +### `` + +Modifies an existing load clause. Supported clauses include: + +- `COLUMNS TERMINATED BY` +- `COLUMNS (...)` +- `PRECEDING FILTER` +- `WHERE` +- `DELETE ON` +- `ORDER BY` +- `PARTITION (...)` + +These clauses can be combined with `PROPERTIES` and `FROM`, but cannot appear in the same statement as `SET TARGET TABLE`. When the target table is switched, Doris reuses the existing load clauses. + +### `` + +Properties in the `PROPERTIES` clause are modified by key. Only explicitly specified keys are changed; omitted keys retain their existing values. + +| Property | Modification requirements | +| --- | --- | +| `desired_concurrent_number` | Must be greater than `0`. Actual concurrency is still limited by the number of BEs, partitions or shards, and cluster configuration. | +| `max_error_number` | Must be greater than or equal to `0`. | +| `max_filter_ratio` | Must be in `[0, 1]`. | +| `max_batch_interval` | Must be greater than or equal to `1`, in seconds. | +| `max_batch_rows` | Must be greater than or equal to `200000`. | +| `max_batch_size` | Must be from 100 MiB through 10 GiB, in bytes. | +| `strict_mode` | `true` or `false`. | +| `timezone` | A valid time zone. It is also used to parse Kafka datetime offsets in the same statement. | +| `workload_group` | A non-empty Workload Group name. | +| `partial_columns` | A backward-compatible property. For Kafka jobs, `true` maps `UPSERT` to `UPDATE_FIXED_COLUMNS`; `false` does not change an existing partial-update mode back to `UPSERT`. Do not specify this property together with `unique_key_update_mode` for a Kinesis job. | +| `unique_key_update_mode` | One of `UPSERT`, `UPDATE_FIXED_COLUMNS`, and `UPDATE_FLEXIBLE_COLUMNS`. For Kafka jobs, this property takes precedence if `partial_columns` is also specified. For Kinesis jobs, use this property alone to disable partial updates or switch modes unambiguously. | +| `jsonpaths` | Changes JSON paths. The file format itself cannot be changed by ALTER. | +| `json_root` | Changes the JSON root. | +| `strip_outer_array` | `true` or `false`. | +| `num_as_string` | `true` or `false`. | +| `fuzzy_parse` | `true` or `false`. | +| `enclose` | This version accepts and records this key, but does not update the enclosing character used by runtime tasks. Do not use ALTER to change it; recreate the job when this character must change. | +| `escape` | This version accepts and records this key, but does not update the escape character used by runtime tasks. Do not use ALTER to change it; recreate the job when this character must change. | +| `empty_field_as_null` | `true` or `false`. | + +Partial updates are also constrained by the target table and the existing load configuration: + +- `UPDATE_FIXED_COLUMNS` requires a Merge-on-Write Unique Key table. +- `UPDATE_FLEXIBLE_COLUMNS` requires a table that supports flexible partial updates. The effective configuration after this ALTER must use JSON, have `fuzzy_parse` disabled, have no non-empty `jsonpaths`, and have no `COLUMNS` mapping. For example, the same statement can set `fuzzy_parse` to `false` or clear `jsonpaths`. +- Multi-table jobs do not support partial-update modes. + +The following create-time properties are not in the ALTER allowlist and cannot be modified by this statement: + +- `format` +- `exec_mem_limit` +- `send_batch_parallelism` +- `load_to_single_tablet` +- `partial_update_new_key_behavior` + +Other unknown properties are also rejected. Property names must use the fixed spelling in the allowlist; use the lowercase names shown in this document. + +### Kafka `` + +A Kafka job must use `FROM KAFKA (...)`. The explicit data source type must match the existing job type; a Kafka job cannot be converted into a Kinesis job. + +| Property | Modification requirements | +| --- | --- | +| `kafka_broker_list` | A list of Kafka broker addresses. Existing values do not need to be repeated during ALTER. Changing only the brokers does not reset progress. | +| `kafka_topic` | A non-empty new topic. Explicitly providing a non-empty value invokes the topic-switch path and resets progress, even when it equals the current topic. An empty value does not clear the current topic. | +| `kafka_partitions` | A comma-separated partition list. It must be specified together with `kafka_offsets` or a default offset. | +| `kafka_offsets` | One value per `kafka_partitions` entry. A value can be a non-negative integer, `OFFSET_BEGINNING`, `OFFSET_END`, or a datetime. Datetime and ordinary offset semantics cannot be mixed in one list. | +| `kafka_default_offsets` | The default offset: `OFFSET_BEGINNING`, `OFFSET_END`, or a datetime. It cannot be specified together with `kafka_offsets`. | +| `property.kafka_default_offsets` | A compatible form of `kafka_default_offsets`. The two forms cannot be specified together. | +| `property.*` | A custom Kafka client property. Doris removes the `property.` prefix internally. A matching key is overwritten and other custom properties are retained. | +| `aws.*` | Doris AWS MSK IAM properties. Related authentication properties must form a complete, valid set in this ALTER statement. | + +Kafka offset changes have these additional requirements: + +- `kafka_partitions` and `kafka_offsets` must contain the same number of items. +- Without a topic switch, only partitions that are already in the job's consumed range can be changed. ALTER cannot add a new consumed partition. +- `kafka_offsets` is mutually exclusive with `kafka_default_offsets` and `property.kafka_default_offsets`. +- `kafka_default_offsets` and `property.kafka_default_offsets` are mutually exclusive. +- A default offset specified without partitions applies to partitions discovered later. When it is specified with `kafka_partitions`, it also initializes the listed partitions. +- If no default offset is explicitly specified, Doris retains the current default instead of synthesizing `OFFSET_END`. + +The following Kafka multi-table routing properties are create-only and cannot be modified by ALTER: + +- `kafka_table_name_location` +- `kafka_table_name_format` +- `kafka_text_table_name_field_delimiter` +- `kafka_text_table_name_field_index` + +When modifying AWS MSK IAM settings, the `FROM KAFKA` clause must contain a self-consistent configuration. For example, it must provide `aws.region`, `property.security.protocol=SASL_SSL`, and `property.sasl.mechanism=OAUTHBEARER`. An external ID requires a role ARN. If the same ALTER statement specifies a public broker and explicit credentials, it must provide both an access key and a secret key. + +### Kinesis `` + +A Kinesis job must use `FROM KINESIS (...)`. The explicit data source type must match the existing job type; a Kinesis job cannot be converted into a Kafka job. Kinesis jobs support the property modifications below, but do not support `SET TARGET TABLE`. + +| Property | Modification requirements | +| --- | --- | +| `aws.region` | An AWS region such as `us-east-1`. The existing value does not need to be repeated during ALTER. | +| `aws.endpoint` | A custom Kinesis endpoint. The legacy key `kinesis_endpoint` is also accepted. | +| `kinesis_stream` | A non-empty new stream. Explicitly providing a non-empty value invokes the stream-switch path and resets progress and discovered shard state, even when it equals the current stream. An empty value does not clear the current stream. | +| `kinesis_shards` | A comma-separated list of shard IDs. It must be specified together with `kinesis_shards_pos` or a default position. | +| `kinesis_shards_pos` | One value per `kinesis_shards` entry. A value can be `TRIM_HORIZON`, `LATEST`, or a numeric sequence number. | +| `property.kinesis_default_pos` | The default position: `TRIM_HORIZON`, `LATEST`, or a numeric sequence number. The `property.` prefix is required. | +| `aws.access_key` | An AWS access key. It must be provided together with `aws.secret_key`. | +| `aws.secret_key` | An AWS secret key. It must be provided together with `aws.access_key`. | +| `aws.session_key` | An optional AWS session token. | +| `aws.role_arn` | An optional IAM role ARN. | +| `property.*` | A custom Kinesis client property. A matching key is overwritten and other custom properties are retained. | + +Kinesis position changes have these additional requirements: + +- `kinesis_shards` and `kinesis_shards_pos` must contain the same number of items. +- Without a stream switch, only positions of currently consumed shards can be changed. +- `kinesis_shards_pos` and `property.kinesis_default_pos` are mutually exclusive. +- Kinesis positions do not accept datetime values. +- A default position specified without shards applies to shards discovered later. When it is specified with `kinesis_shards`, it also initializes the listed shards. +- If a `FROM KINESIS` ALTER specifies neither `kinesis_shards_pos` nor `property.kinesis_default_pos`, Doris synthesizes `LATEST` and records it as the new default position. To retain a non-`LATEST` default while changing only the region, endpoint, stream, credentials, or another custom property, repeat the current `property.kinesis_default_pos` in the same statement. +- The access key and secret key must be provided together in the same ALTER statement. `property.aws.external.id` also requires a role ARN. + +## Property Modification Semantics + +`job_properties`, Kafka properties, and Kinesis properties generally use key-based incremental update semantics: + +- Only explicitly specified keys are changed and omitted keys retain their current values, except for the Kinesis default-position behavior described above. +- Custom `property.*` values are merged with the existing properties, and a matching key is overwritten. +- There is no general `UNSET` or property-deletion syntax. Omitting a key does not delete it. +- Authentication properties with dependencies should be provided as a complete set in the same ALTER statement. +- Do not repeat an unchanged, non-empty `kafka_topic` or `kinesis_stream`, because doing so triggers the corresponding source-switch and progress-reset behavior. ## Access Control Requirements -Users executing this SQL command must have at least the following privileges: +The user executing this command needs at least the following privileges: | Privilege | Object | Notes | -| :-------- | :----- | :---- | -| LOAD_PRIV | Table | ALTER ROUTINE LOAD requires LOAD privilege on the table | +| --- | --- | --- | +| `LOAD_PRIV` | Current target table | Modifying a Routine Load job requires `LOAD_PRIV` on its current target table. | +| `LOAD_PRIV` | New target table | `SET TARGET TABLE` additionally requires `LOAD_PRIV` on the new target table. | -## Notes +## Progress and Data Behavior -- `kafka_partitions` and `kafka_offsets` are used to modify the offset of kafka partitions to be consumed, and can only modify currently consumed partitions. New partitions cannot be added. +- Switching only the target table of a Kafka job does not reset Kafka offsets. Existing rows remain in the old table; after the job resumes, newly consumed data is written to the new table. +- Changing ordinary job properties, Kafka brokers, the Kinesis region or endpoint, or custom `property.*` values does not reset progress. +- Explicitly providing a non-empty `kafka_topic` resets Kafka progress. +- Explicitly providing a non-empty `kinesis_stream` resets Kinesis progress, discovered shards, and the lag cache. +- If one statement switches the target table and also provides a non-empty `kafka_topic`, progress is reset because of the topic change. It is not a target-only switch. ## Examples -- Modify `desired_concurrent_number` to 1 - - ```sql - ALTER ROUTINE LOAD FOR db1.label1 - PROPERTIES - ( - "desired_concurrent_number" = "1" - ); - ``` - -- Modify `desired_concurrent_number` to 10, modify partition offsets, and modify group id - - ```sql - ALTER ROUTINE LOAD FOR db1.label1 - PROPERTIES - ( - "desired_concurrent_number" = "10" - ) - FROM kafka - ( - "kafka_partitions" = "0, 1, 2", - "kafka_offsets" = "100, 200, 100", - "property.group.id" = "new_group" - ); - ``` \ No newline at end of file +### Switch the target table of a Kafka job and retain progress + +In this example, `db1.job1` is a single-table Kafka Routine Load job. + +```sql +PAUSE ROUTINE LOAD FOR db1.job1; + +ALTER ROUTINE LOAD FOR db1.job1 +SET TARGET TABLE = "new_table"; + +RESUME ROUTINE LOAD FOR db1.job1; +``` + +Already consumed rows remain in the old table. After the job resumes, new rows are written to `new_table`. + +### Switch the target table and modify properties together + +This example switches the target table, changes the error threshold, and updates the Kafka client ID. Kafka progress is retained because `kafka_topic` is not explicitly specified. + +```sql +PAUSE ROUTINE LOAD FOR db1.job1; + +ALTER ROUTINE LOAD FOR db1.job1 +SET TARGET TABLE = "new_table" +PROPERTIES ( + "max_error_number" = "10" +) +FROM KAFKA ( + "property.client.id" = "target-switch" +); + +RESUME ROUTINE LOAD FOR db1.job1; +``` + +### Change Kafka partition offsets + +```sql +PAUSE ROUTINE LOAD FOR db1.job1; + +ALTER ROUTINE LOAD FOR db1.job1 +FROM KAFKA ( + "kafka_partitions" = "0,1,2", + "kafka_offsets" = "100,200,100" +); + +RESUME ROUTINE LOAD FOR db1.job1; +``` + +### Change Kinesis shard positions + +```sql +PAUSE ROUTINE LOAD FOR db1.kinesis_job; + +ALTER ROUTINE LOAD FOR db1.kinesis_job +FROM KINESIS ( + "kinesis_shards" = "shardId-000000000000,shardId-000000000001", + "kinesis_shards_pos" = "TRIM_HORIZON,LATEST" +); + +RESUME ROUTINE LOAD FOR db1.kinesis_job; +``` + +After the modification, use `SHOW ROUTINE LOAD FOR ` to inspect the target table, job properties, data source properties, and consumption progress. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md index 6d05564b98e60..b9f2d1ca4655b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md @@ -2,105 +2,273 @@ { "title": "ALTER ROUTINE LOAD", "language": "zh-CN", - "description": "该语法用于修改已经创建的例行导入作业。只能修改处于 PAUSED 状态的作业。" + "description": "修改已创建的 Routine Load 作业，说明作业属性、Kafka 与 Kinesis 数据源属性的修改要求、消费进度行为，以及仅单表 Kafka 作业支持的目标表切换语法、权限、表兼容性和使用限制。" } --- ## 描述 -该语法用于修改已经创建的例行导入作业。只能修改处于 PAUSED 状态的作业。 +`ALTER ROUTINE LOAD` 用于修改已创建的 Routine Load 作业。对于单表 Kafka 作业，可以在一条语句中切换目标表，并同时修改支持的作业属性和 Kafka 数据源属性。Kinesis 作业支持修改作业属性和数据源属性，但不支持切换目标表。 + +只能修改处于 `PAUSED` 状态的作业。推荐按照 `PAUSE ROUTINE LOAD`、`ALTER ROUTINE LOAD`、`RESUME ROUTINE LOAD` 的顺序执行。 ## 语法 ```sql -ALTER ROUTINE LOAD FOR [.] -[] -FROM [] -[] +ALTER ROUTINE LOAD FOR [.] +[ + SET TARGET TABLE = "" + | [, ...] +] +[PROPERTIES ( + "" = "" + [, ...] +)] +[FROM ( + "" = "" + [, ...] +)]; ``` +语句中至少需要指定一项修改。 + +指定 `SET TARGET TABLE` 时，原作业必须是 Kafka 作业，可选的 `FROM` 子句也必须是 `FROM KAFKA`。Kinesis 作业可以使用其他 ALTER 形式，但不能指定 `SET TARGET TABLE`。 + ## 必选参数 -**1. `[.]`** +### `[.]` + +指定要修改的 Routine Load 作业。 -> 指定要修改的作业名称。标识符必须以字母字符开头，并且不能包含空格或特殊字符，除非整个标识符字符串用反引号括起来。 -> -> 标识符不能使用保留关键字。有关更多详细信息，请参阅标识符要求和保留关键字。 +- 如果省略 ``，Doris 使用当前数据库。 +- 作业名称可以使用反引号包围。 ## 可选参数 -**1. ``** - -> 指定需要修改的作业参数。目前支持修改的参数包括： -> -> - desired_concurrent_number -> - max_error_number -> - max_batch_interval -> - max_batch_rows -> - max_batch_size -> - jsonpaths -> - json_root -> - strip_outer_array -> - strict_mode -> - timezone -> - num_as_string -> - fuzzy_parse -> - partial_columns -> - max_filter_ratio - -**2. ``** - -> 数据源的相关属性。目前支持： -> -> - kafka_partitions -> - kafka_offsets -> - kafka_broker_list -> - kafka_topic -> - 自定义 property，如 property.group.id - -**3. ``** - -> 数据源的类型。当前支持： -> -> - KAFKA +### `SET TARGET TABLE` + +将单表 Kafka Routine Load 作业的目标表切换为 ``。 + +目标表名称必须写成字符串字面量，例如 `SET TARGET TABLE = "new_table"`。不支持旧的 `ON new_table` 写法，也不支持省略引号的 `SET TARGET TABLE = new_table`。 + +目标表必须满足以下要求： + +- 与 Routine Load 作业位于同一数据库。`` 中不能指定其他数据库。 +- 是非临时 OLAP 表。 +- 作业的数据源类型是 Kafka。Kinesis 作业不支持切换目标表。 +- 作业是单表 Routine Load 作业；不支持 multi-table 作业。 +- 当前用户对原目标表和新目标表都具有 `LOAD_PRIV` 权限。 +- 新表兼容作业已有的列映射、过滤条件、固定分区、合并或删除条件、sequence 列以及部分列更新模式。Doris 会使用已有导入配置在新表上重新生成导入计划，无法生成计划时拒绝修改。 +- 如果作业创建时启用了 `load_to_single_tablet=true`，新表必须使用 Random Distribution。 + +`SET TARGET TABLE` 可以与 `PROPERTIES` 和 `FROM KAFKA` 组合使用，但不能与 `FROM KINESIS` 或 `` 组合使用。 + +### `` + +修改已有的导入子句。支持的子句包括： + +- `COLUMNS TERMINATED BY` +- `COLUMNS (...)` +- `PRECEDING FILTER` +- `WHERE` +- `DELETE ON` +- `ORDER BY` +- `PARTITION (...)` + +这些子句可以与 `PROPERTIES` 和 `FROM` 组合使用，但不能与 `SET TARGET TABLE` 出现在同一条语句中。切换目标表时，Doris 会直接复用原作业的这些导入子句。 + +### `` + +在 `PROPERTIES` 子句中按键修改 Routine Load 作业属性。仅修改本次显式指定的键，未指定的键保持原值。 + +| 属性 | 修改要求 | +| --- | --- | +| `desired_concurrent_number` | 大于 `0`。实际并发数仍受 BE 数量、分区或 shard 数量以及集群配置限制。 | +| `max_error_number` | 大于等于 `0`。 | +| `max_filter_ratio` | 取值范围为 `[0, 1]`。 | +| `max_batch_interval` | 大于等于 `1`，单位为秒。 | +| `max_batch_rows` | 大于等于 `200000`。 | +| `max_batch_size` | 取值范围为 100 MiB 到 10 GiB，单位为字节。 | +| `strict_mode` | `true` 或 `false`。 | +| `timezone` | 合法的时区名称。该时区也用于解析同一条语句中的 Kafka datetime offset。 | +| `workload_group` | 非空的 Workload Group 名称。 | +| `partial_columns` | 兼容旧版本的属性。对于 Kafka 作业，`true` 会把 `UPSERT` 映射为 `UPDATE_FIXED_COLUMNS`；`false` 不会把已有的部分更新模式切回 `UPSERT`。对于 Kinesis 作业，不要同时指定本属性和 `unique_key_update_mode`。 | +| `unique_key_update_mode` | 仅支持 `UPSERT`、`UPDATE_FIXED_COLUMNS`、`UPDATE_FLEXIBLE_COLUMNS`。对于 Kafka 作业，如果同时指定 `partial_columns`，本属性优先。对于 Kinesis 作业，关闭部分更新或切换模式时只使用本属性，以避免歧义。 | +| `jsonpaths` | 修改 JSON 路径。不能通过 ALTER 修改作业的文件格式。 | +| `json_root` | 修改 JSON 根节点。 | +| `strip_outer_array` | `true` 或 `false`。 | +| `num_as_string` | `true` 或 `false`。 | +| `fuzzy_parse` | `true` 或 `false`。 | +| `enclose` | 当前版本会接受并记录此键，但不会更新运行时任务使用的 CSV 包围符。不要通过 ALTER 修改该字符；需要修改时应重建作业。 | +| `escape` | 当前版本会接受并记录此键，但不会更新运行时任务使用的 CSV 转义符。不要通过 ALTER 修改该字符；需要修改时应重建作业。 | +| `empty_field_as_null` | `true` 或 `false`。 | + +部分列更新还受目标表和已有导入配置约束： + +- `UPDATE_FIXED_COLUMNS` 要求目标表为 Merge-on-Write 的 Unique Key 表。 +- `UPDATE_FLEXIBLE_COLUMNS` 要求目标表支持灵活部分列更新，且应用本次 ALTER 后的有效配置必须使用 JSON 格式、未启用 `fuzzy_parse`、未配置非空 `jsonpaths`，也未配置 `COLUMNS` 映射。例如，可以在同一条语句中把 `fuzzy_parse` 改为 `false` 或清空 `jsonpaths`。 +- multi-table 作业不支持部分列更新模式。 + +以下创建期属性不在 ALTER 白名单中，不能通过本语句修改： + +- `format` +- `exec_mem_limit` +- `send_batch_parallelism` +- `load_to_single_tablet` +- `partial_update_new_key_behavior` + +其他未知属性也会被拒绝。属性名按白名单进行大小写敏感匹配，必须使用文档列出的全小写名称。 + +### Kafka `` + +Kafka 作业必须使用 `FROM KAFKA (...)`。显式数据源类型必须与原作业一致，不能把 Kafka 作业转换为 Kinesis 作业。 + +| 属性 | 修改要求 | +| --- | --- | +| `kafka_broker_list` | Kafka broker 地址列表。ALTER 时不需要重复提供原值。仅修改 broker 不重置消费进度。 | +| `kafka_topic` | 非空的新 topic。只要在 ALTER 中显式提供非空值，Doris 就会按 topic 切换流程重置消费进度，即使值与原值相同。空值不会清除原 topic。 | +| `kafka_partitions` | 逗号分隔的 partition 列表。必须与 `kafka_offsets` 或默认 offset 同时指定。 | +| `kafka_offsets` | 与 `kafka_partitions` 一一对应。每项可为非负整数、`OFFSET_BEGINNING`、`OFFSET_END` 或 datetime。同一列表不能混用 datetime 和普通 offset。 | +| `kafka_default_offsets` | 默认 offset，可取 `OFFSET_BEGINNING`、`OFFSET_END` 或 datetime。不能与 `kafka_offsets` 同时指定。 | +| `property.kafka_default_offsets` | `kafka_default_offsets` 的兼容写法；两种写法不能同时使用。 | +| `property.*` | 传递给 Kafka 客户端的自定义属性。`property.` 前缀在内部会被移除；同名键覆盖，其他已有自定义属性保留。 | +| `aws.*` | Doris 的 AWS MSK IAM 属性命名空间。相关认证属性必须在本次 ALTER 中满足依赖关系。 | + +Kafka offset 修改还需要满足以下要求： + +- `kafka_partitions` 与 `kafka_offsets` 的项目数量必须一致。 +- 不切换 topic 时，只能修改作业当前已消费范围内的 partition，不能通过 ALTER 增加新的消费 partition。 +- `kafka_offsets` 与 `kafka_default_offsets` 或 `property.kafka_default_offsets` 互斥。 +- `kafka_default_offsets` 与 `property.kafka_default_offsets` 互斥。 +- 单独修改默认 offset 时，新值用于后续新发现的 partition；与 `kafka_partitions` 一起指定时，该默认值同时用于列出的 partition。 +- 未显式指定默认 offset 时，Doris 保留原默认值，不会自动写入 `OFFSET_END`。 + +以下 Kafka multi-table 路由属性只能在创建作业时设置，不能通过 ALTER 修改： + +- `kafka_table_name_location` +- `kafka_table_name_format` +- `kafka_text_table_name_field_delimiter` +- `kafka_text_table_name_field_index` + +修改 AWS MSK IAM 配置时，本次 `FROM KAFKA` 子句需要提供一组自洽的配置。例如，需要提供 `aws.region`、`property.security.protocol=SASL_SSL` 和 `property.sasl.mechanism=OAUTHBEARER`；使用 external ID 时还需要 role ARN；本次 ALTER 同时指定公网 broker 和显式凭据时，access key 和 secret key 必须成对提供。 + +### Kinesis `` + +Kinesis 作业必须使用 `FROM KINESIS (...)`。显式数据源类型必须与原作业一致，不能把 Kinesis 作业转换为 Kafka 作业。Kinesis 作业支持下列属性修改，但不支持 `SET TARGET TABLE`。 + +| 属性 | 修改要求 | +| --- | --- | +| `aws.region` | AWS region，例如 `us-east-1`。ALTER 时不需要重复提供原值。 | +| `aws.endpoint` | 自定义 Kinesis endpoint。兼容旧键 `kinesis_endpoint`。 | +| `kinesis_stream` | 非空的新 stream。只要在 ALTER 中显式提供非空值，Doris 就会按 stream 切换流程重置消费进度和已发现的 shard 状态，即使值与原值相同。空值不会清除原 stream。 | +| `kinesis_shards` | 逗号分隔的 shard ID。必须与 `kinesis_shards_pos` 或默认 position 同时指定。 | +| `kinesis_shards_pos` | 与 `kinesis_shards` 一一对应。每项可为 `TRIM_HORIZON`、`LATEST` 或数字 sequence number。 | +| `property.kinesis_default_pos` | 默认 position，可为 `TRIM_HORIZON`、`LATEST` 或数字 sequence number。必须保留 `property.` 前缀。 | +| `aws.access_key` | AWS access key。与 `aws.secret_key` 成对提供。 | +| `aws.secret_key` | AWS secret key。与 `aws.access_key` 成对提供。 | +| `aws.session_key` | 可选的 AWS session token。 | +| `aws.role_arn` | 可选的 IAM role ARN。 | +| `property.*` | 传递给 Kinesis 客户端的自定义属性。同名键覆盖，其他已有自定义属性保留。 | + +Kinesis position 修改还需要满足以下要求： + +- `kinesis_shards` 与 `kinesis_shards_pos` 的项目数量必须一致。 +- 不切换 stream 时，只能修改当前已消费 shard 的 position。 +- `kinesis_shards_pos` 与 `property.kinesis_default_pos` 互斥。 +- Kinesis position 不支持 datetime。 +- 单独修改默认 position 时，新值用于后续新发现的 shard；与 `kinesis_shards` 一起指定时，该默认值同时用于列出的 shard。 +- 如果一条 `FROM KINESIS` ALTER 既未指定 `kinesis_shards_pos`，也未指定 `property.kinesis_default_pos`，Doris 会合成 `LATEST` 并将其记录为新的默认 position。仅修改 region、endpoint、stream、凭据或其他自定义属性时，如果需要保留一个非 `LATEST` 的默认值，必须在同一条语句中重复提供当前的 `property.kinesis_default_pos`。 +- access key 与 secret key 必须在同一条 ALTER 中成对提供；使用 `property.aws.external.id` 时还需要提供 role ARN。 + +## 属性修改语义 + +`job_properties`、Kafka properties 和 Kinesis properties 通常采用按键增量修改语义： + +- 只修改本次显式提供的键，未提供的键保持原值，但上文所述的 Kinesis 默认 position 行为除外。 +- `property.*` 自定义属性与现有属性合并；同名键覆盖。 +- 当前没有通用的 `UNSET` 或删除属性语法。省略一个键不代表删除它。 +- 如果需要修改具有依赖关系的认证属性，应在同一条 ALTER 中提供完整的一组属性。 +- 不要重复提交没有变化的非空 `kafka_topic` 或 `kinesis_stream`，否则会触发对应的数据源切换和进度重置。 ## 权限控制 -执行此 SQL 命令的用户必须至少具有以下权限： +执行此命令的用户至少需要以下权限： -| 权限（Privilege） | 对象（Object） | 说明（Notes） | -| :---------------- | :------------- | :---------------------------- | -| LOAD_PRIV | 表（Table） | SHOW ROUTINE LOAD 需要对表有LOAD权限 | +| 权限 | 对象 | 说明 | +| --- | --- | --- | +| `LOAD_PRIV` | 当前目标表 | 修改 Routine Load 作业需要对当前目标表具有 `LOAD_PRIV` 权限。 | +| `LOAD_PRIV` | 新目标表 | 使用 `SET TARGET TABLE` 时，还需要对新目标表具有 `LOAD_PRIV` 权限。 | -## 注意事项 +## 进度和数据行为 -- `kafka_partitions` 和 `kafka_offsets` 用于修改待消费的 kafka partition 的 offset，仅能修改当前已经消费的 partition。不能新增 partition。 +- 仅切换 Kafka 作业的目标表不会重置 Kafka offset。旧表中的历史数据不会迁移；恢复作业后，新消费的数据写入新表。 +- 修改普通作业属性、Kafka broker、Kinesis region 或 endpoint、以及自定义 `property.*` 不会重置消费进度。 +- 显式提供非空 `kafka_topic` 会重置 Kafka progress。 +- 显式提供非空 `kinesis_stream` 会重置 Kinesis progress、已发现的 shard 和 lag cache。 +- 如果同一条语句同时切换目标表并提供非空 `kafka_topic`，进度会因为 topic 切换而重置，不能视为纯目标表切换。 ## 示例 -- 将 `desired_concurrent_number` 修改为 1 - - ```sql - ALTER ROUTINE LOAD FOR db1.label1 - PROPERTIES - ( - "desired_concurrent_number" = "1" - ); - ``` - -- 将 `desired_concurrent_number` 修改为 10，修改 partition 的 offset，修改 group id - - ```sql - ALTER ROUTINE LOAD FOR db1.label1 - PROPERTIES - ( - "desired_concurrent_number" = "10" - ) - FROM kafka - ( - "kafka_partitions" = "0, 1, 2", - "kafka_offsets" = "100, 200, 100", - "property.group.id" = "new_group" - ); - ``` +### 切换 Kafka 作业的目标表并保留消费进度 + +本例中的 `db1.job1` 是单表 Kafka Routine Load 作业。 + +```sql +PAUSE ROUTINE LOAD FOR db1.job1; + +ALTER ROUTINE LOAD FOR db1.job1 +SET TARGET TABLE = "new_table"; + +RESUME ROUTINE LOAD FOR db1.job1; +``` + +已消费的数据仍保留在原表中。恢复后，新数据写入 `new_table`。 + +### 同时切换目标表和修改属性 + +以下示例切换目标表、修改错误阈值，并更新 Kafka client ID。由于没有显式修改 `kafka_topic`，Kafka progress 保持不变。 + +```sql +PAUSE ROUTINE LOAD FOR db1.job1; + +ALTER ROUTINE LOAD FOR db1.job1 +SET TARGET TABLE = "new_table" +PROPERTIES ( + "max_error_number" = "10" +) +FROM KAFKA ( + "property.client.id" = "target-switch" +); + +RESUME ROUTINE LOAD FOR db1.job1; +``` + +### 修改 Kafka partition offset + +```sql +PAUSE ROUTINE LOAD FOR db1.job1; + +ALTER ROUTINE LOAD FOR db1.job1 +FROM KAFKA ( + "kafka_partitions" = "0,1,2", + "kafka_offsets" = "100,200,100" +); + +RESUME ROUTINE LOAD FOR db1.job1; +``` + +### 修改 Kinesis shard position + +```sql +PAUSE ROUTINE LOAD FOR db1.kinesis_job; + +ALTER ROUTINE LOAD FOR db1.kinesis_job +FROM KINESIS ( + "kinesis_shards" = "shardId-000000000000,shardId-000000000001", + "kinesis_shards_pos" = "TRIM_HORIZON,LATEST" +); + +RESUME ROUTINE LOAD FOR db1.kinesis_job; +``` +修改后可以使用 `SHOW ROUTINE LOAD FOR ` 检查目标表、作业属性、数据源属性和消费进度。 From 3466ea87bfb21b1c2ab6edbdab7305215cacc380 Mon Sep 17 00:00:00 2001 From: Refrain Date: Thu, 16 Jul 2026 15:09:26 +0800 Subject: [PATCH 2/3] docs: simplify ALTER ROUTINE LOAD reference --- .../load-and-export/ALTER-ROUTINE-LOAD.md | 278 ++++-------------- .../load-and-export/ALTER-ROUTINE-LOAD.md | 278 ++++-------------- 2 files changed, 130 insertions(+), 426 deletions(-) diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md index 10f683b343e65..9842a25b62aef 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md @@ -2,273 +2,125 @@ { "title": "ALTER ROUTINE LOAD", "language": "en", - "description": "Modifies an existing Routine Load job. Kafka jobs can also switch their target table." + "description": "Modifies a paused Routine Load job, including its Kafka target table, job properties, and Kafka source properties." } --- ## Description -`ALTER ROUTINE LOAD` modifies an existing Routine Load job. For a single-table Kafka job, one statement can switch the target table and modify supported job and Kafka data source properties at the same time. Kinesis jobs support job and data source property modifications, but do not support target-table switching. +This statement modifies an existing Routine Load job. Only jobs in the `PAUSED` state can be modified. You can pause a Routine Load job by using [PAUSE ROUTINE LOAD](./PAUSE-ROUTINE-LOAD.md). -Only a job in the `PAUSED` state can be modified. The recommended sequence is `PAUSE ROUTINE LOAD`, `ALTER ROUTINE LOAD`, and then `RESUME ROUTINE LOAD`. +After a successful modification, you can: + +- Use [SHOW ROUTINE LOAD](./SHOW-ROUTINE-LOAD.md) to check the modified job details. +- Use [RESUME ROUTINE LOAD](./RESUME-ROUTINE-LOAD.md) to restart the job. ## Syntax ```sql -ALTER ROUTINE LOAD FOR [.] -[ - SET TARGET TABLE = "" - | [, ...] -] +ALTER ROUTINE LOAD FOR [.] +[SET TARGET TABLE = ""] [PROPERTIES ( "" = "" [, ...] )] -[FROM ( +[FROM ( "" = "" [, ...] )]; ``` -The statement must specify at least one modification. - -When `SET TARGET TABLE` is specified, the existing job must be a Kafka job and an optional `FROM` clause must be `FROM KAFKA`. A Kinesis job can use the other ALTER forms but cannot specify `SET TARGET TABLE`. - -## Required Parameters - -### `[.]` - -Specifies the Routine Load job to modify. - -- If `` is omitted, Doris uses the current database. -- The job name can be enclosed in backticks. - -## Optional Parameters - -### `SET TARGET TABLE` - -Switches the target table of a single-table Kafka Routine Load job to ``. +## Parameters -The target table name must be a string literal, for example, `SET TARGET TABLE = "new_table"`. The old `ON new_table` form and an unquoted `SET TARGET TABLE = new_table` are not supported. +### 1. `[.]` -The target table must meet all of the following requirements: +Specifies the name of the job to modify. An identifier must start with a letter and cannot contain spaces or special characters unless the entire identifier is enclosed in backticks. -- It is in the same database as the Routine Load job. `` cannot specify another database. -- It is a non-temporary OLAP table. -- The job uses Kafka as its data source. Kinesis jobs do not support target-table switching. -- The job is a single-table Routine Load job. Multi-table jobs are not supported. -- The current user has the `LOAD_PRIV` privilege on both the old and new target tables. -- The new table is compatible with the existing column mappings, filters, fixed partitions, merge or delete conditions, sequence column, and partial-update mode. Doris replans the existing load configuration against the new table and rejects the change if a valid load plan cannot be generated. -- If the job was created with `load_to_single_tablet=true`, the new table uses Random Distribution. +An identifier cannot use a reserved keyword. For more information, see [Identifier Requirements](../../../basic-element/object-identifiers.md) and [Reserved Keywords](../../../basic-element/reserved-keywords.md). -`SET TARGET TABLE` can be combined with `PROPERTIES` and `FROM KAFKA`, but it cannot be combined with `FROM KINESIS` or ``. +### 2. `SET TARGET TABLE = ""` -### `` +Specifies the new target table for the load job. -Modifies an existing load clause. Supported clauses include: +Switching only the target table does not move historical data. Existing batches remain in the old table. After the job resumes, Doris continues consuming from the retained Kafka offsets and writes new batches to the new table. -- `COLUMNS TERMINATED BY` -- `COLUMNS (...)` -- `PRECEDING FILTER` -- `WHERE` -- `DELETE ON` -- `ORDER BY` -- `PARTITION (...)` +The target table must be in the same database as the job and must be a non-temporary OLAP table. Target-table switching supports only single-table Kafka Routine Load jobs. -These clauses can be combined with `PROPERTIES` and `FROM`, but cannot appear in the same statement as `SET TARGET TABLE`. When the target table is switched, Doris reuses the existing load clauses. +### 3. `` -### `` +Specifies the job properties to modify. The following properties are currently supported: -Properties in the `PROPERTIES` clause are modified by key. Only explicitly specified keys are changed; omitted keys retain their existing values. +- `desired_concurrent_number` +- `max_error_number` +- `max_batch_interval` +- `max_batch_rows` +- `max_batch_size` +- `jsonpaths` +- `json_root` +- `strip_outer_array` +- `strict_mode` +- `timezone` +- `num_as_string` +- `fuzzy_parse` +- `partial_columns` +- `max_filter_ratio` -| Property | Modification requirements | -| --- | --- | -| `desired_concurrent_number` | Must be greater than `0`. Actual concurrency is still limited by the number of BEs, partitions or shards, and cluster configuration. | -| `max_error_number` | Must be greater than or equal to `0`. | -| `max_filter_ratio` | Must be in `[0, 1]`. | -| `max_batch_interval` | Must be greater than or equal to `1`, in seconds. | -| `max_batch_rows` | Must be greater than or equal to `200000`. | -| `max_batch_size` | Must be from 100 MiB through 10 GiB, in bytes. | -| `strict_mode` | `true` or `false`. | -| `timezone` | A valid time zone. It is also used to parse Kafka datetime offsets in the same statement. | -| `workload_group` | A non-empty Workload Group name. | -| `partial_columns` | A backward-compatible property. For Kafka jobs, `true` maps `UPSERT` to `UPDATE_FIXED_COLUMNS`; `false` does not change an existing partial-update mode back to `UPSERT`. Do not specify this property together with `unique_key_update_mode` for a Kinesis job. | -| `unique_key_update_mode` | One of `UPSERT`, `UPDATE_FIXED_COLUMNS`, and `UPDATE_FLEXIBLE_COLUMNS`. For Kafka jobs, this property takes precedence if `partial_columns` is also specified. For Kinesis jobs, use this property alone to disable partial updates or switch modes unambiguously. | -| `jsonpaths` | Changes JSON paths. The file format itself cannot be changed by ALTER. | -| `json_root` | Changes the JSON root. | -| `strip_outer_array` | `true` or `false`. | -| `num_as_string` | `true` or `false`. | -| `fuzzy_parse` | `true` or `false`. | -| `enclose` | This version accepts and records this key, but does not update the enclosing character used by runtime tasks. Do not use ALTER to change it; recreate the job when this character must change. | -| `escape` | This version accepts and records this key, but does not update the escape character used by runtime tasks. Do not use ALTER to change it; recreate the job when this character must change. | -| `empty_field_as_null` | `true` or `false`. | +### 4. `` -Partial updates are also constrained by the target table and the existing load configuration: +Specifies the data source type. When used together with `SET TARGET TABLE`, currently only the following type is supported: -- `UPDATE_FIXED_COLUMNS` requires a Merge-on-Write Unique Key table. -- `UPDATE_FLEXIBLE_COLUMNS` requires a table that supports flexible partial updates. The effective configuration after this ALTER must use JSON, have `fuzzy_parse` disabled, have no non-empty `jsonpaths`, and have no `COLUMNS` mapping. For example, the same statement can set `fuzzy_parse` to `false` or clear `jsonpaths`. -- Multi-table jobs do not support partial-update modes. +- `KAFKA` -The following create-time properties are not in the ALTER allowlist and cannot be modified by this statement: +### 5. `` -- `format` -- `exec_mem_limit` -- `send_batch_parallelism` -- `load_to_single_tablet` -- `partial_update_new_key_behavior` +Specifies the data source properties to modify. The following properties are currently supported: -Other unknown properties are also rejected. Property names must use the fixed spelling in the allowlist; use the lowercase names shown in this document. - -### Kafka `` - -A Kafka job must use `FROM KAFKA (...)`. The explicit data source type must match the existing job type; a Kafka job cannot be converted into a Kinesis job. - -| Property | Modification requirements | -| --- | --- | -| `kafka_broker_list` | A list of Kafka broker addresses. Existing values do not need to be repeated during ALTER. Changing only the brokers does not reset progress. | -| `kafka_topic` | A non-empty new topic. Explicitly providing a non-empty value invokes the topic-switch path and resets progress, even when it equals the current topic. An empty value does not clear the current topic. | -| `kafka_partitions` | A comma-separated partition list. It must be specified together with `kafka_offsets` or a default offset. | -| `kafka_offsets` | One value per `kafka_partitions` entry. A value can be a non-negative integer, `OFFSET_BEGINNING`, `OFFSET_END`, or a datetime. Datetime and ordinary offset semantics cannot be mixed in one list. | -| `kafka_default_offsets` | The default offset: `OFFSET_BEGINNING`, `OFFSET_END`, or a datetime. It cannot be specified together with `kafka_offsets`. | -| `property.kafka_default_offsets` | A compatible form of `kafka_default_offsets`. The two forms cannot be specified together. | -| `property.*` | A custom Kafka client property. Doris removes the `property.` prefix internally. A matching key is overwritten and other custom properties are retained. | -| `aws.*` | Doris AWS MSK IAM properties. Related authentication properties must form a complete, valid set in this ALTER statement. | - -Kafka offset changes have these additional requirements: - -- `kafka_partitions` and `kafka_offsets` must contain the same number of items. -- Without a topic switch, only partitions that are already in the job's consumed range can be changed. ALTER cannot add a new consumed partition. -- `kafka_offsets` is mutually exclusive with `kafka_default_offsets` and `property.kafka_default_offsets`. -- `kafka_default_offsets` and `property.kafka_default_offsets` are mutually exclusive. -- A default offset specified without partitions applies to partitions discovered later. When it is specified with `kafka_partitions`, it also initializes the listed partitions. -- If no default offset is explicitly specified, Doris retains the current default instead of synthesizing `OFFSET_END`. - -The following Kafka multi-table routing properties are create-only and cannot be modified by ALTER: - -- `kafka_table_name_location` -- `kafka_table_name_format` -- `kafka_text_table_name_field_delimiter` -- `kafka_text_table_name_field_index` - -When modifying AWS MSK IAM settings, the `FROM KAFKA` clause must contain a self-consistent configuration. For example, it must provide `aws.region`, `property.security.protocol=SASL_SSL`, and `property.sasl.mechanism=OAUTHBEARER`. An external ID requires a role ARN. If the same ALTER statement specifies a public broker and explicit credentials, it must provide both an access key and a secret key. - -### Kinesis `` - -A Kinesis job must use `FROM KINESIS (...)`. The explicit data source type must match the existing job type; a Kinesis job cannot be converted into a Kafka job. Kinesis jobs support the property modifications below, but do not support `SET TARGET TABLE`. - -| Property | Modification requirements | -| --- | --- | -| `aws.region` | An AWS region such as `us-east-1`. The existing value does not need to be repeated during ALTER. | -| `aws.endpoint` | A custom Kinesis endpoint. The legacy key `kinesis_endpoint` is also accepted. | -| `kinesis_stream` | A non-empty new stream. Explicitly providing a non-empty value invokes the stream-switch path and resets progress and discovered shard state, even when it equals the current stream. An empty value does not clear the current stream. | -| `kinesis_shards` | A comma-separated list of shard IDs. It must be specified together with `kinesis_shards_pos` or a default position. | -| `kinesis_shards_pos` | One value per `kinesis_shards` entry. A value can be `TRIM_HORIZON`, `LATEST`, or a numeric sequence number. | -| `property.kinesis_default_pos` | The default position: `TRIM_HORIZON`, `LATEST`, or a numeric sequence number. The `property.` prefix is required. | -| `aws.access_key` | An AWS access key. It must be provided together with `aws.secret_key`. | -| `aws.secret_key` | An AWS secret key. It must be provided together with `aws.access_key`. | -| `aws.session_key` | An optional AWS session token. | -| `aws.role_arn` | An optional IAM role ARN. | -| `property.*` | A custom Kinesis client property. A matching key is overwritten and other custom properties are retained. | - -Kinesis position changes have these additional requirements: - -- `kinesis_shards` and `kinesis_shards_pos` must contain the same number of items. -- Without a stream switch, only positions of currently consumed shards can be changed. -- `kinesis_shards_pos` and `property.kinesis_default_pos` are mutually exclusive. -- Kinesis positions do not accept datetime values. -- A default position specified without shards applies to shards discovered later. When it is specified with `kinesis_shards`, it also initializes the listed shards. -- If a `FROM KINESIS` ALTER specifies neither `kinesis_shards_pos` nor `property.kinesis_default_pos`, Doris synthesizes `LATEST` and records it as the new default position. To retain a non-`LATEST` default while changing only the region, endpoint, stream, credentials, or another custom property, repeat the current `property.kinesis_default_pos` in the same statement. -- The access key and secret key must be provided together in the same ALTER statement. `property.aws.external.id` also requires a role ARN. - -## Property Modification Semantics - -`job_properties`, Kafka properties, and Kinesis properties generally use key-based incremental update semantics: - -- Only explicitly specified keys are changed and omitted keys retain their current values, except for the Kinesis default-position behavior described above. -- Custom `property.*` values are merged with the existing properties, and a matching key is overwritten. -- There is no general `UNSET` or property-deletion syntax. Omitting a key does not delete it. -- Authentication properties with dependencies should be provided as a complete set in the same ALTER statement. -- Do not repeat an unchanged, non-empty `kafka_topic` or `kinesis_stream`, because doing so triggers the corresponding source-switch and progress-reset behavior. +- `kafka_broker_list` +- `kafka_topic` +- Custom properties, such as `property.group.id` +- `kafka_partitions` and `kafka_offsets`, which modify the offsets of Kafka partitions to be consumed. When `kafka_topic` is not changed, only currently consumed partitions can be modified; new partitions cannot be added. ## Access Control Requirements -The user executing this command needs at least the following privileges: +The user executing this SQL command must have at least the following privileges: | Privilege | Object | Notes | | --- | --- | --- | -| `LOAD_PRIV` | Current target table | Modifying a Routine Load job requires `LOAD_PRIV` on its current target table. | -| `LOAD_PRIV` | New target table | `SET TARGET TABLE` additionally requires `LOAD_PRIV` on the new target table. | - -## Progress and Data Behavior - -- Switching only the target table of a Kafka job does not reset Kafka offsets. Existing rows remain in the old table; after the job resumes, newly consumed data is written to the new table. -- Changing ordinary job properties, Kafka brokers, the Kinesis region or endpoint, or custom `property.*` values does not reset progress. -- Explicitly providing a non-empty `kafka_topic` resets Kafka progress. -- Explicitly providing a non-empty `kinesis_stream` resets Kinesis progress, discovered shards, and the lag cache. -- If one statement switches the target table and also provides a non-empty `kafka_topic`, progress is reset because of the topic change. It is not a target-only switch. +| `LOAD_PRIV` | Current target table | Required when modifying a single-table Routine Load job. | +| `LOAD_PRIV` | Database | Required when modifying a multi-table Routine Load job. | +| `LOAD_PRIV` | New target table | Additionally required when using `SET TARGET TABLE`. | ## Examples -### Switch the target table of a Kafka job and retain progress - -In this example, `db1.job1` is a single-table Kafka Routine Load job. +### Modify `desired_concurrent_number` to `1` ```sql -PAUSE ROUTINE LOAD FOR db1.job1; - -ALTER ROUTINE LOAD FOR db1.job1 -SET TARGET TABLE = "new_table"; - -RESUME ROUTINE LOAD FOR db1.job1; -``` - -Already consumed rows remain in the old table. After the job resumes, new rows are written to `new_table`. - -### Switch the target table and modify properties together - -This example switches the target table, changes the error threshold, and updates the Kafka client ID. Kafka progress is retained because `kafka_topic` is not explicitly specified. - -```sql -PAUSE ROUTINE LOAD FOR db1.job1; - -ALTER ROUTINE LOAD FOR db1.job1 -SET TARGET TABLE = "new_table" -PROPERTIES ( - "max_error_number" = "10" -) -FROM KAFKA ( - "property.client.id" = "target-switch" +ALTER ROUTINE LOAD FOR db1.label1 +PROPERTIES +( + "desired_concurrent_number" = "1" ); - -RESUME ROUTINE LOAD FOR db1.job1; ``` -### Change Kafka partition offsets +### Modify `desired_concurrent_number`, partition offsets, and group ID ```sql -PAUSE ROUTINE LOAD FOR db1.job1; - -ALTER ROUTINE LOAD FOR db1.job1 -FROM KAFKA ( - "kafka_partitions" = "0,1,2", - "kafka_offsets" = "100,200,100" +ALTER ROUTINE LOAD FOR db1.label1 +PROPERTIES +( + "desired_concurrent_number" = "10" +) +FROM KAFKA +( + "kafka_partitions" = "0, 1, 2", + "kafka_offsets" = "100, 200, 100", + "property.group.id" = "new_group" ); - -RESUME ROUTINE LOAD FOR db1.job1; ``` -### Change Kinesis shard positions +### Switch the target table to `new_table_name` ```sql -PAUSE ROUTINE LOAD FOR db1.kinesis_job; - -ALTER ROUTINE LOAD FOR db1.kinesis_job -FROM KINESIS ( - "kinesis_shards" = "shardId-000000000000,shardId-000000000001", - "kinesis_shards_pos" = "TRIM_HORIZON,LATEST" -); - -RESUME ROUTINE LOAD FOR db1.kinesis_job; +ALTER ROUTINE LOAD FOR db1.label1 +SET TARGET TABLE = "new_table_name"; ``` - -After the modification, use `SHOW ROUTINE LOAD FOR ` to inspect the target table, job properties, data source properties, and consumption progress. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md index b9f2d1ca4655b..13dbd1989b68a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md @@ -2,273 +2,125 @@ { "title": "ALTER ROUTINE LOAD", "language": "zh-CN", - "description": "修改已创建的 Routine Load 作业，说明作业属性、Kafka 与 Kinesis 数据源属性的修改要求、消费进度行为，以及仅单表 Kafka 作业支持的目标表切换语法、权限、表兼容性和使用限制。" + "description": "修改已暂停的 Routine Load 作业，说明 Kafka 单表作业切换目标表、修改作业属性、更新 Kafka 数据源属性与消费 offset 的语法、权限要求和使用示例。" } --- ## 描述 -`ALTER ROUTINE LOAD` 用于修改已创建的 Routine Load 作业。对于单表 Kafka 作业，可以在一条语句中切换目标表，并同时修改支持的作业属性和 Kafka 数据源属性。Kinesis 作业支持修改作业属性和数据源属性，但不支持切换目标表。 +该语法用于修改已经创建的 Routine Load 导入作业。只能修改处于 `PAUSED` 状态的作业。您可以通过 [PAUSE ROUTINE LOAD](./PAUSE-ROUTINE-LOAD.md) 暂停 Routine Load 导入作业。 -只能修改处于 `PAUSED` 状态的作业。推荐按照 `PAUSE ROUTINE LOAD`、`ALTER ROUTINE LOAD`、`RESUME ROUTINE LOAD` 的顺序执行。 +修改成功后，您可以： + +- 通过 [SHOW ROUTINE LOAD](./SHOW-ROUTINE-LOAD.md) 检查修改后的作业详情。 +- 通过 [RESUME ROUTINE LOAD](./RESUME-ROUTINE-LOAD.md) 重启该导入作业。 ## 语法 ```sql -ALTER ROUTINE LOAD FOR [.] -[ - SET TARGET TABLE = "" - | [, ...] -] +ALTER ROUTINE LOAD FOR [.] +[SET TARGET TABLE = ""] [PROPERTIES ( "" = "" [, ...] )] -[FROM ( +[FROM ( "" = "" [, ...] )]; ``` -语句中至少需要指定一项修改。 - -指定 `SET TARGET TABLE` 时，原作业必须是 Kafka 作业，可选的 `FROM` 子句也必须是 `FROM KAFKA`。Kinesis 作业可以使用其他 ALTER 形式，但不能指定 `SET TARGET TABLE`。 - -## 必选参数 - -### `[.]` - -指定要修改的 Routine Load 作业。 - -- 如果省略 ``，Doris 使用当前数据库。 -- 作业名称可以使用反引号包围。 - -## 可选参数 - -### `SET TARGET TABLE` - -将单表 Kafka Routine Load 作业的目标表切换为 ``。 +## 参数说明 -目标表名称必须写成字符串字面量，例如 `SET TARGET TABLE = "new_table"`。不支持旧的 `ON new_table` 写法，也不支持省略引号的 `SET TARGET TABLE = new_table`。 +### 1. `[.]` -目标表必须满足以下要求： +指定要修改的作业名称。标识符必须以字母字符开头，并且不能包含空格或特殊字符，除非整个标识符字符串用反引号括起来。 -- 与 Routine Load 作业位于同一数据库。`` 中不能指定其他数据库。 -- 是非临时 OLAP 表。 -- 作业的数据源类型是 Kafka。Kinesis 作业不支持切换目标表。 -- 作业是单表 Routine Load 作业；不支持 multi-table 作业。 -- 当前用户对原目标表和新目标表都具有 `LOAD_PRIV` 权限。 -- 新表兼容作业已有的列映射、过滤条件、固定分区、合并或删除条件、sequence 列以及部分列更新模式。Doris 会使用已有导入配置在新表上重新生成导入计划，无法生成计划时拒绝修改。 -- 如果作业创建时启用了 `load_to_single_tablet=true`，新表必须使用 Random Distribution。 +标识符不能使用保留关键字。有关更多详细信息，请参阅[标识符要求](../../../basic-element/object-identifiers.md)和[保留关键字](../../../basic-element/reserved-keywords.md)。 -`SET TARGET TABLE` 可以与 `PROPERTIES` 和 `FROM KAFKA` 组合使用，但不能与 `FROM KINESIS` 或 `` 组合使用。 +### 2. `SET TARGET TABLE = ""` -### `` +指定要切换到的导入目标表。 -修改已有的导入子句。支持的子句包括： +纯切表不会搬迁历史数据。旧批次仍保留在旧表中；作业恢复后，Doris 从保留的 Kafka offset 继续消费，并把新批次写入新表。 -- `COLUMNS TERMINATED BY` -- `COLUMNS (...)` -- `PRECEDING FILTER` -- `WHERE` -- `DELETE ON` -- `ORDER BY` -- `PARTITION (...)` +目标表必须与作业位于同一数据库，且为非临时 OLAP 表。只有单表 Kafka Routine Load 作业支持切换目标表。 -这些子句可以与 `PROPERTIES` 和 `FROM` 组合使用，但不能与 `SET TARGET TABLE` 出现在同一条语句中。切换目标表时，Doris 会直接复用原作业的这些导入子句。 +### 3. `` -### `` +指定需要修改的作业参数。目前支持修改的参数包括： -在 `PROPERTIES` 子句中按键修改 Routine Load 作业属性。仅修改本次显式指定的键，未指定的键保持原值。 +- `desired_concurrent_number` +- `max_error_number` +- `max_batch_interval` +- `max_batch_rows` +- `max_batch_size` +- `jsonpaths` +- `json_root` +- `strip_outer_array` +- `strict_mode` +- `timezone` +- `num_as_string` +- `fuzzy_parse` +- `partial_columns` +- `max_filter_ratio` -| 属性 | 修改要求 | -| --- | --- | -| `desired_concurrent_number` | 大于 `0`。实际并发数仍受 BE 数量、分区或 shard 数量以及集群配置限制。 | -| `max_error_number` | 大于等于 `0`。 | -| `max_filter_ratio` | 取值范围为 `[0, 1]`。 | -| `max_batch_interval` | 大于等于 `1`，单位为秒。 | -| `max_batch_rows` | 大于等于 `200000`。 | -| `max_batch_size` | 取值范围为 100 MiB 到 10 GiB，单位为字节。 | -| `strict_mode` | `true` 或 `false`。 | -| `timezone` | 合法的时区名称。该时区也用于解析同一条语句中的 Kafka datetime offset。 | -| `workload_group` | 非空的 Workload Group 名称。 | -| `partial_columns` | 兼容旧版本的属性。对于 Kafka 作业，`true` 会把 `UPSERT` 映射为 `UPDATE_FIXED_COLUMNS`；`false` 不会把已有的部分更新模式切回 `UPSERT`。对于 Kinesis 作业，不要同时指定本属性和 `unique_key_update_mode`。 | -| `unique_key_update_mode` | 仅支持 `UPSERT`、`UPDATE_FIXED_COLUMNS`、`UPDATE_FLEXIBLE_COLUMNS`。对于 Kafka 作业，如果同时指定 `partial_columns`，本属性优先。对于 Kinesis 作业，关闭部分更新或切换模式时只使用本属性，以避免歧义。 | -| `jsonpaths` | 修改 JSON 路径。不能通过 ALTER 修改作业的文件格式。 | -| `json_root` | 修改 JSON 根节点。 | -| `strip_outer_array` | `true` 或 `false`。 | -| `num_as_string` | `true` 或 `false`。 | -| `fuzzy_parse` | `true` 或 `false`。 | -| `enclose` | 当前版本会接受并记录此键，但不会更新运行时任务使用的 CSV 包围符。不要通过 ALTER 修改该字符；需要修改时应重建作业。 | -| `escape` | 当前版本会接受并记录此键，但不会更新运行时任务使用的 CSV 转义符。不要通过 ALTER 修改该字符；需要修改时应重建作业。 | -| `empty_field_as_null` | `true` 或 `false`。 | +### 4. `` -部分列更新还受目标表和已有导入配置约束： +数据源的类型。与 `SET TARGET TABLE` 组合使用时，当前仅支持： -- `UPDATE_FIXED_COLUMNS` 要求目标表为 Merge-on-Write 的 Unique Key 表。 -- `UPDATE_FLEXIBLE_COLUMNS` 要求目标表支持灵活部分列更新，且应用本次 ALTER 后的有效配置必须使用 JSON 格式、未启用 `fuzzy_parse`、未配置非空 `jsonpaths`，也未配置 `COLUMNS` 映射。例如，可以在同一条语句中把 `fuzzy_parse` 改为 `false` 或清空 `jsonpaths`。 -- multi-table 作业不支持部分列更新模式。 +- `KAFKA` -以下创建期属性不在 ALTER 白名单中，不能通过本语句修改： +### 5. `` -- `format` -- `exec_mem_limit` -- `send_batch_parallelism` -- `load_to_single_tablet` -- `partial_update_new_key_behavior` +数据源的相关属性。目前仅支持： -其他未知属性也会被拒绝。属性名按白名单进行大小写敏感匹配，必须使用文档列出的全小写名称。 - -### Kafka `` - -Kafka 作业必须使用 `FROM KAFKA (...)`。显式数据源类型必须与原作业一致，不能把 Kafka 作业转换为 Kinesis 作业。 - -| 属性 | 修改要求 | -| --- | --- | -| `kafka_broker_list` | Kafka broker 地址列表。ALTER 时不需要重复提供原值。仅修改 broker 不重置消费进度。 | -| `kafka_topic` | 非空的新 topic。只要在 ALTER 中显式提供非空值，Doris 就会按 topic 切换流程重置消费进度，即使值与原值相同。空值不会清除原 topic。 | -| `kafka_partitions` | 逗号分隔的 partition 列表。必须与 `kafka_offsets` 或默认 offset 同时指定。 | -| `kafka_offsets` | 与 `kafka_partitions` 一一对应。每项可为非负整数、`OFFSET_BEGINNING`、`OFFSET_END` 或 datetime。同一列表不能混用 datetime 和普通 offset。 | -| `kafka_default_offsets` | 默认 offset，可取 `OFFSET_BEGINNING`、`OFFSET_END` 或 datetime。不能与 `kafka_offsets` 同时指定。 | -| `property.kafka_default_offsets` | `kafka_default_offsets` 的兼容写法；两种写法不能同时使用。 | -| `property.*` | 传递给 Kafka 客户端的自定义属性。`property.` 前缀在内部会被移除；同名键覆盖，其他已有自定义属性保留。 | -| `aws.*` | Doris 的 AWS MSK IAM 属性命名空间。相关认证属性必须在本次 ALTER 中满足依赖关系。 | - -Kafka offset 修改还需要满足以下要求： - -- `kafka_partitions` 与 `kafka_offsets` 的项目数量必须一致。 -- 不切换 topic 时，只能修改作业当前已消费范围内的 partition，不能通过 ALTER 增加新的消费 partition。 -- `kafka_offsets` 与 `kafka_default_offsets` 或 `property.kafka_default_offsets` 互斥。 -- `kafka_default_offsets` 与 `property.kafka_default_offsets` 互斥。 -- 单独修改默认 offset 时，新值用于后续新发现的 partition；与 `kafka_partitions` 一起指定时，该默认值同时用于列出的 partition。 -- 未显式指定默认 offset 时，Doris 保留原默认值，不会自动写入 `OFFSET_END`。 - -以下 Kafka multi-table 路由属性只能在创建作业时设置，不能通过 ALTER 修改： - -- `kafka_table_name_location` -- `kafka_table_name_format` -- `kafka_text_table_name_field_delimiter` -- `kafka_text_table_name_field_index` - -修改 AWS MSK IAM 配置时，本次 `FROM KAFKA` 子句需要提供一组自洽的配置。例如，需要提供 `aws.region`、`property.security.protocol=SASL_SSL` 和 `property.sasl.mechanism=OAUTHBEARER`；使用 external ID 时还需要 role ARN；本次 ALTER 同时指定公网 broker 和显式凭据时，access key 和 secret key 必须成对提供。 - -### Kinesis `` - -Kinesis 作业必须使用 `FROM KINESIS (...)`。显式数据源类型必须与原作业一致，不能把 Kinesis 作业转换为 Kafka 作业。Kinesis 作业支持下列属性修改，但不支持 `SET TARGET TABLE`。 - -| 属性 | 修改要求 | -| --- | --- | -| `aws.region` | AWS region，例如 `us-east-1`。ALTER 时不需要重复提供原值。 | -| `aws.endpoint` | 自定义 Kinesis endpoint。兼容旧键 `kinesis_endpoint`。 | -| `kinesis_stream` | 非空的新 stream。只要在 ALTER 中显式提供非空值，Doris 就会按 stream 切换流程重置消费进度和已发现的 shard 状态，即使值与原值相同。空值不会清除原 stream。 | -| `kinesis_shards` | 逗号分隔的 shard ID。必须与 `kinesis_shards_pos` 或默认 position 同时指定。 | -| `kinesis_shards_pos` | 与 `kinesis_shards` 一一对应。每项可为 `TRIM_HORIZON`、`LATEST` 或数字 sequence number。 | -| `property.kinesis_default_pos` | 默认 position，可为 `TRIM_HORIZON`、`LATEST` 或数字 sequence number。必须保留 `property.` 前缀。 | -| `aws.access_key` | AWS access key。与 `aws.secret_key` 成对提供。 | -| `aws.secret_key` | AWS secret key。与 `aws.access_key` 成对提供。 | -| `aws.session_key` | 可选的 AWS session token。 | -| `aws.role_arn` | 可选的 IAM role ARN。 | -| `property.*` | 传递给 Kinesis 客户端的自定义属性。同名键覆盖，其他已有自定义属性保留。 | - -Kinesis position 修改还需要满足以下要求： - -- `kinesis_shards` 与 `kinesis_shards_pos` 的项目数量必须一致。 -- 不切换 stream 时，只能修改当前已消费 shard 的 position。 -- `kinesis_shards_pos` 与 `property.kinesis_default_pos` 互斥。 -- Kinesis position 不支持 datetime。 -- 单独修改默认 position 时，新值用于后续新发现的 shard；与 `kinesis_shards` 一起指定时，该默认值同时用于列出的 shard。 -- 如果一条 `FROM KINESIS` ALTER 既未指定 `kinesis_shards_pos`，也未指定 `property.kinesis_default_pos`，Doris 会合成 `LATEST` 并将其记录为新的默认 position。仅修改 region、endpoint、stream、凭据或其他自定义属性时，如果需要保留一个非 `LATEST` 的默认值，必须在同一条语句中重复提供当前的 `property.kinesis_default_pos`。 -- access key 与 secret key 必须在同一条 ALTER 中成对提供；使用 `property.aws.external.id` 时还需要提供 role ARN。 - -## 属性修改语义 - -`job_properties`、Kafka properties 和 Kinesis properties 通常采用按键增量修改语义： - -- 只修改本次显式提供的键，未提供的键保持原值，但上文所述的 Kinesis 默认 position 行为除外。 -- `property.*` 自定义属性与现有属性合并；同名键覆盖。 -- 当前没有通用的 `UNSET` 或删除属性语法。省略一个键不代表删除它。 -- 如果需要修改具有依赖关系的认证属性，应在同一条 ALTER 中提供完整的一组属性。 -- 不要重复提交没有变化的非空 `kafka_topic` 或 `kinesis_stream`，否则会触发对应的数据源切换和进度重置。 +- `kafka_broker_list` +- `kafka_topic` +- 自定义 `property`，如 `property.group.id` +- `kafka_partitions` 和 `kafka_offsets`，用于修改待消费的 Kafka partition 的 offset。未修改 `kafka_topic` 时，仅能修改当前已经消费的 partition，不能新增 partition。 ## 权限控制 -执行此命令的用户至少需要以下权限： +执行此 SQL 命令的用户必须至少具有以下权限： | 权限 | 对象 | 说明 | | --- | --- | --- | -| `LOAD_PRIV` | 当前目标表 | 修改 Routine Load 作业需要对当前目标表具有 `LOAD_PRIV` 权限。 | -| `LOAD_PRIV` | 新目标表 | 使用 `SET TARGET TABLE` 时，还需要对新目标表具有 `LOAD_PRIV` 权限。 | - -## 进度和数据行为 - -- 仅切换 Kafka 作业的目标表不会重置 Kafka offset。旧表中的历史数据不会迁移；恢复作业后，新消费的数据写入新表。 -- 修改普通作业属性、Kafka broker、Kinesis region 或 endpoint、以及自定义 `property.*` 不会重置消费进度。 -- 显式提供非空 `kafka_topic` 会重置 Kafka progress。 -- 显式提供非空 `kinesis_stream` 会重置 Kinesis progress、已发现的 shard 和 lag cache。 -- 如果同一条语句同时切换目标表并提供非空 `kafka_topic`，进度会因为 topic 切换而重置，不能视为纯目标表切换。 +| `LOAD_PRIV` | 当前目标表 | 修改单表 Routine Load 作业时需要。 | +| `LOAD_PRIV` | 数据库 | 修改 multi-table Routine Load 作业时需要。 | +| `LOAD_PRIV` | 新目标表 | 使用 `SET TARGET TABLE` 时还需要此权限。 | ## 示例 -### 切换 Kafka 作业的目标表并保留消费进度 - -本例中的 `db1.job1` 是单表 Kafka Routine Load 作业。 +### 将 `desired_concurrent_number` 修改为 `1` ```sql -PAUSE ROUTINE LOAD FOR db1.job1; - -ALTER ROUTINE LOAD FOR db1.job1 -SET TARGET TABLE = "new_table"; - -RESUME ROUTINE LOAD FOR db1.job1; -``` - -已消费的数据仍保留在原表中。恢复后，新数据写入 `new_table`。 - -### 同时切换目标表和修改属性 - -以下示例切换目标表、修改错误阈值，并更新 Kafka client ID。由于没有显式修改 `kafka_topic`，Kafka progress 保持不变。 - -```sql -PAUSE ROUTINE LOAD FOR db1.job1; - -ALTER ROUTINE LOAD FOR db1.job1 -SET TARGET TABLE = "new_table" -PROPERTIES ( - "max_error_number" = "10" -) -FROM KAFKA ( - "property.client.id" = "target-switch" +ALTER ROUTINE LOAD FOR db1.label1 +PROPERTIES +( + "desired_concurrent_number" = "1" ); - -RESUME ROUTINE LOAD FOR db1.job1; ``` -### 修改 Kafka partition offset +### 将 `desired_concurrent_number` 修改为 `10`，并修改 partition offset 和 group ID ```sql -PAUSE ROUTINE LOAD FOR db1.job1; - -ALTER ROUTINE LOAD FOR db1.job1 -FROM KAFKA ( - "kafka_partitions" = "0,1,2", - "kafka_offsets" = "100,200,100" +ALTER ROUTINE LOAD FOR db1.label1 +PROPERTIES +( + "desired_concurrent_number" = "10" +) +FROM KAFKA +( + "kafka_partitions" = "0, 1, 2", + "kafka_offsets" = "100, 200, 100", + "property.group.id" = "new_group" ); - -RESUME ROUTINE LOAD FOR db1.job1; ``` -### 修改 Kinesis shard position +### 将目标表修改为 `new_table_name` ```sql -PAUSE ROUTINE LOAD FOR db1.kinesis_job; - -ALTER ROUTINE LOAD FOR db1.kinesis_job -FROM KINESIS ( - "kinesis_shards" = "shardId-000000000000,shardId-000000000001", - "kinesis_shards_pos" = "TRIM_HORIZON,LATEST" -); - -RESUME ROUTINE LOAD FOR db1.kinesis_job; +ALTER ROUTINE LOAD FOR db1.label1 +SET TARGET TABLE = "new_table_name"; ``` - -修改后可以使用 `SHOW ROUTINE LOAD FOR ` 检查目标表、作业属性、数据源属性和消费进度。 From c6517651b71102f121fbe46c646da437c366893f Mon Sep 17 00:00:00 2001 From: Refrain Date: Thu, 16 Jul 2026 15:30:05 +0800 Subject: [PATCH 3/3] docs: update Japanese ALTER ROUTINE LOAD reference --- .../load-and-export/ALTER-ROUTINE-LOAD.md | 184 ++++++++++-------- 1 file changed, 104 insertions(+), 80 deletions(-) diff --git a/ja-source/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md b/ja-source/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md index c0dd42e0ef24f..6e97a07dd94f3 100644 --- a/ja-source/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md +++ b/ja-source/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/ALTER-ROUTINE-LOAD.md @@ -2,101 +2,125 @@ { "title": "ALTER ROUTINE LOAD", "language": "ja", - "description": "この構文は既存のルーチンロードジョブを変更するために使用されます。PAUSED状態のジョブのみ変更可能です。" + "description": "一時停止中の Routine Load ジョブを変更する構文について、単一テーブルの Kafka Routine Load ジョブのターゲットテーブル切り替え、ジョブプロパティ、Kafka データソースプロパティ、権限、および使用例を説明します。" } --- + ## 説明 -この構文は、既存のroutine loadジョブを変更するために使用されます。PAUSED状態のジョブのみ変更できます。 +この構文は、作成済みの Routine Load ジョブを変更するために使用します。`PAUSED` 状態のジョブのみ変更できます。[PAUSE ROUTINE LOAD](./PAUSE-ROUTINE-LOAD.md) を使用して、Routine Load ジョブを一時停止できます。 + +変更が成功した後は、次の操作を実行できます。 + +- [SHOW ROUTINE LOAD](./SHOW-ROUTINE-LOAD.md) を使用して、変更後のジョブの詳細を確認します。 +- [RESUME ROUTINE LOAD](./RESUME-ROUTINE-LOAD.md) を使用して、ジョブを再開します。 ## 構文 ```sql ALTER ROUTINE LOAD FOR [.] -[] -FROM [] -[] +[SET TARGET TABLE = ""] +[PROPERTIES ( + "" = "" + [, ...] +)] +[FROM ( + "" = "" + [, ...] +)]; ``` -## 必須パラメータ - -**1. `[.]`** - -> 変更するジョブの名前を指定します。識別子は文字で始まる必要があり、識別子文字列全体がバッククォートで囲まれていない限り、スペースや特殊文字を含めることはできません。 -> -> 識別子は予約キーワードを使用できません。詳細については、識別子要件と予約キーワードを参照してください。 - -## オプションパラメータ - -**1. ``** - -> 変更するジョブパラメータを指定します。現在サポートされているパラメータには以下が含まれます： -> -> - desired_concurrent_number -> - max_error_number -> - max_batch_interval -> - max_batch_rows -> - max_batch_size -> - jsonpaths -> - json_root -> - strip_outer_array -> - strict_mode -> - timezone -> - num_as_string -> - fuzzy_parse -> - partial_columns -> - max_filter_ratio - -**2. ``** - -> データソースに関連するプロパティ。現在サポートされているもの： -> -> - `` -> - `` -> - `` -> - `` -> - ``などのカスタムプロパティ - -**3. ``** - -> データソースのタイプ。現在サポートされているもの： -> -> - KAFKA -## アクセス制御要件 +## パラメータ + +### 1. `[.]` + +変更するジョブ名を指定します。識別子は英字で始める必要があり、識別子全体をバッククォートで囲まない限り、空白または特殊文字を含めることはできません。 + +予約キーワードは識別子として使用できません。詳細については、[識別子の要件](../../../basic-element/object-identifiers.md)および[予約キーワード](../../../basic-element/reserved-keywords.md)を参照してください。 + +### 2. `SET TARGET TABLE = ""` + +切り替え先となる新しいターゲットテーブルを指定します。 + +ターゲットテーブルだけを切り替えても、過去のデータは移動されません。既存のバッチは旧テーブルに残ります。ジョブの再開後、Doris は保持されている Kafka オフセットから消費を続行し、新しいバッチを新しいテーブルに書き込みます。 + +ターゲットテーブルは、ジョブと同じデータベースに属する、一時テーブルではない OLAP テーブルである必要があります。ターゲットテーブルの切り替えは、単一テーブルの Kafka Routine Load ジョブでのみサポートされます。 + +### 3. `` + +変更するジョブプロパティを指定します。現在、次のプロパティがサポートされています。 + +- `desired_concurrent_number` +- `max_error_number` +- `max_batch_interval` +- `max_batch_rows` +- `max_batch_size` +- `jsonpaths` +- `json_root` +- `strip_outer_array` +- `strict_mode` +- `timezone` +- `num_as_string` +- `fuzzy_parse` +- `partial_columns` +- `max_filter_ratio` + +### 4. `` + +データソースの種類を指定します。`SET TARGET TABLE` と併用する場合、現在サポートされているのは次の種類のみです。 -このSQLコマンドを実行するユーザーは、少なくとも以下の権限を持つ必要があります： +- `KAFKA` -| 権限 | オブジェクト | 備考 | -| :-------- | :----- | :---- | -| LOAD_PRIV | テーブル | SHOW ROUTINE LOADはテーブルに対するLOAD権限が必要 | +### 5. `` -## 注意事項 +変更するデータソースプロパティを指定します。現在、次のプロパティがサポートされています。 -- `kafka_partitions`と`kafka_offsets`は、消費されるkafkaパーティションのオフセットを変更するために使用され、現在消費されているパーティションのみを変更できます。新しいパーティションを追加することはできません。 +- `kafka_broker_list` +- `kafka_topic` +- `property.group.id` などのカスタムプロパティ +- `kafka_partitions` と `kafka_offsets` は、消費対象の Kafka パーティションのオフセットを変更するために使用します。`kafka_topic` を変更しない場合、現在消費しているパーティションのオフセットのみ変更でき、新しいパーティションは追加できません。 + +## アクセス制御要件 + +この SQL コマンドを実行するユーザーには、少なくとも次の権限が必要です。 + +| 権限 | オブジェクト | 説明 | +| --- | --- | --- | +| `LOAD_PRIV` | 現在のターゲットテーブル | 単一テーブルの Routine Load ジョブを変更する場合に必要です。 | +| `LOAD_PRIV` | データベース | マルチテーブルの Routine Load ジョブを変更する場合に必要です。 | +| `LOAD_PRIV` | 新しいターゲットテーブル | `SET TARGET TABLE` を使用する場合に追加で必要です。 | ## 例 -- `desired_concurrent_number`を1に変更 - - ```sql - ALTER ROUTINE LOAD FOR db1.label1 - PROPERTIES - ( - "desired_concurrent_number" = "1" - ); - ``` -- `desired_concurrent_number`を10に変更し、パーティションオフセットを変更し、グループIDを変更する - - ```sql - ALTER ROUTINE LOAD FOR db1.label1 - PROPERTIES - ( - "desired_concurrent_number" = "10" - ) - FROM kafka - ( - "kafka_partitions" = "0, 1, 2", - "kafka_offsets" = "100, 200, 100", - "property.group.id" = "new_group" - ); - ``` +### `desired_concurrent_number` を `1` に変更する + +```sql +ALTER ROUTINE LOAD FOR db1.label1 +PROPERTIES +( + "desired_concurrent_number" = "1" +); +``` + +### `desired_concurrent_number` を `10` に変更し、パーティションのオフセットとグループ ID も変更する + +```sql +ALTER ROUTINE LOAD FOR db1.label1 +PROPERTIES +( + "desired_concurrent_number" = "10" +) +FROM KAFKA +( + "kafka_partitions" = "0, 1, 2", + "kafka_offsets" = "100, 200, 100", + "property.group.id" = "new_group" +); +``` + +### ターゲットテーブルを `new_table_name` に切り替える + +```sql +ALTER ROUTINE LOAD FOR db1.label1 +SET TARGET TABLE = "new_table_name"; +```