Antalya 26.1: Remote initiator improvements#1577
Antalya 26.1: Remote initiator improvements#1577ianton-ru wants to merge 10 commits intoantalya-26.1from
Conversation
|
@codex review |
Audit: PR #1577 — Antalya 26.1: Remote initiator improvementsSource: Altinity/ClickHouse#1577 Confirmed defectsNone. No confirmed defects in reviewed scope. (Earlier revision briefly introduced an unused Coverage summary
Audit metadata
AI-generated audit note. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0d1972000
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value; | ||
| if (remote_initiator_cluster_name.empty()) | ||
| remote_initiator_cluster_name = cluster_name_from_settings; | ||
| auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name); | ||
| auto storage_and_context = convertToRemote(remote_initiator_cluster, context, remote_initiator_cluster_name, query_to_send); |
There was a problem hiding this comment.
When object_storage_remote_initiator_cluster is set, the query forwarded via remote(...) still carries that setting even though convertToRemote only clears object_storage_remote_initiator. This makes remote execution depend on remote nodes understanding a setting that is only needed on the initiator; in mixed-version/rolling-upgrade clusters, older remote hosts can fail with unknown-setting errors before execution. The forwarded AST settings should drop object_storage_remote_initiator_cluster together with object_storage_remote_initiator.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@ianton-ru does it make sense? it looks like it does
There was a problem hiding this comment.
object_storage_remote_initiator_cluster does nothing without ``object_storage_remote_initiator`. Make sense to remove just for less garbage in sub-query.
| Prewhere filter | ||
| Prewhere filter column: less(multiply(2, b), 100) | ||
| Filter column: and(indexHint(greater(plus(i, 40), 0)), equals(a, 0)) (removed) | ||
| Filter column: and(equals(a, 0), indexHint(greater(plus(i, 40), 0))) (removed) |
There was a problem hiding this comment.
Argument order depends on hash, hash was changes (see FunctionNode::updateTreeHashImpl)
| if (settings[Setting::object_storage_remote_initiator]) | ||
| { | ||
| auto storage_and_context = convertToRemote(cluster, context, cluster_name_from_settings, query_to_send); | ||
| auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value; |
There was a problem hiding this comment.
Please add a comment explaining what this code block does. It took me a while to understand it by just reading the code.
I suggest something like:
/// In case the current node is not supposed to initiate the clustered query
/// Sends this query to a remote initiator using the `remote` table function
if (settings[Setting::object_storage_remote_initiator])
{
/// Re-writes queries in the form of:
/// Input: SELECT * FROM iceberg(...) SETTINGS object_storage_cluster='swarm', object_storage_remote_initiator=1
/// Output: SELECT * FROM remote('remote_host', icebergCluster('swarm', ...)
/// Where `remote_host` is a random host from the cluster which will execute the query
/// This means the initiator node belongs to the same cluster that will execute the query
/// In case remote_initiator_cluster_name is set, the initiator might be set to a different cluster
}
| if (shard_addresses.size() != 1) | ||
| throw Exception(ErrorCodes::LOGICAL_ERROR, "Size of shard {} in cluster {} is not equal 1", shard_num, cluster_name_from_settings); | ||
| auto host_name = shard_addresses[0].toString(); | ||
| std::string host_name; |
There was a problem hiding this comment.
Address here in encoded format foo%2Ebar instead of foo.bar. This wasn't catched in tests before I add a host with dot in name.
| auto remote_query = makeASTFunction(remote_function_name, make_intrusive<ASTLiteral>(host_name), table_expression->table_function); | ||
| boost::intrusive_ptr<ASTFunction> remote_query; | ||
|
|
||
| if (shard_addresses[0].user_specified) |
| throw Exception(ErrorCodes::CLUSTER_DOESNT_EXIST, "Requested cluster '{}' not found", cluster_name); | ||
| /// Remove check cluster existing here | ||
| /// In query like | ||
| /// remote('remote_host', xxxCluster('remote_cluster', ...)) |
There was a problem hiding this comment.
What if the query is not remote? Can't we check for that?
/// If cluster not exists, query falls later
Where and with which exception? It would be good to avoid any network calls before failing
There was a problem hiding this comment.
Cluster name is not a network node name, it's an internal ClickHouse name. Query falls later when tries to get hosts from cluster. Network calls can't be made without hosts.
But it's hard to understand here is cluster function inside 'remote' or not.
| auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value; | ||
| if (remote_initiator_cluster_name.empty()) | ||
| remote_initiator_cluster_name = cluster_name_from_settings; | ||
| auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name); | ||
| auto storage_and_context = convertToRemote(remote_initiator_cluster, context, remote_initiator_cluster_name, query_to_send); |
There was a problem hiding this comment.
@ianton-ru does it make sense? it looks like it does
|
The changes look ok, but I think it needs more documentation. I also wonder if we can keep the 'cluster exists' check by wrapping that check in a `if (!is_remote0' |
Audit Report: PR #1577 — Remote initiator improvementsScope: Altinity/ClickHouse PR #1577 AI audit note: This review comment was generated by AI (gpt-5.3-codex). Confirmed defectsMedium: Forwarded query does not strip initiator-only setting
Coverage summary
|
PR #1577 CI Verification ReportCI Results Overview
PR's New Test ValidationThe PR adds new integration tests for Latest run (Mar 30):
All 13 test executions passed on the latest CI run across amd_binary, amd_asan, and arm_binary configurations. CI Failures1.
|
| Suite | x86_64 | aarch64 |
|---|---|---|
| Iceberg (1) | Fail (3h30m timeout) | Pass |
| Iceberg (2) | Pass | Pass |
| Parquet | Pass | Pass |
| Parquet (aws_s3) | Pass | Pass |
| Parquet (minio) | Pass | Pass |
| S3 Export (part) | Pass | Pass |
| S3 Export (partition) | Pass | Pass |
| Swarms | Fail | Fail |
Regression Failure: Iceberg (1) x86_64 — Pre-existing Timeout
The Iceberg 1 suite hit the 3h30m job timeout on x86_64. Database analysis (30-day window) shows this is a pre-existing issue:
- 26.1.6.20001.altinityantalya x86_64: 2 Fail / 11 OK (~15% fail rate)
- 26.1.4.20001.altinityantalya x86_64: 7 Fail / 51 OK (~12% fail rate)
- 26.1.3.20001.altinityantalya x86_64: 66 Fail / 18 OK (~79% fail rate)
- Iceberg 1 aarch64 passed. Iceberg 2 passed on both architectures.
Assessment: Flaky (pre-existing) — Intermittent timeout affecting the Iceberg 1 suite, not specific to this PR.
Regression Failure: Swarms — Pre-existing Node Failure Instability
Two failing scenarios on both x86_64 and aarch64:
/swarms/feature/node failure/initiator out of disk space (Fail)
Database check for: /swarms/feature/node failure/initiator out of disk space
- History: 52/113 fails in last 7 days, 202/423 fails in last 30 days
- Last fail: 2026-03-30
- Last pass: 2026-03-30
- Concentration: All versions affected — 26.1.6 (73%), 26.1.4 (28%), 26.1.3 (78%), 25.8.16 (5%), 25.8.14 (57%)
- Error signature: Consistent — UNKNOWN_DATABASE exception (Code: 81)
Assessment: Flaky (pre-existing)
Recommendation: Known unstable test, no action needed for PR verification
/swarms/feature/node failure/check restart clickhouse on swarm node (Error — 600s timeout)
Database check for: /swarms/feature/node failure/check restart clickhouse on swarm node
- History: 266 Fail + 60 Error / 97 OK in last 30 days (~77% failure rate)
- Last fail: 2026-03-26
- Last pass: 2026-03-30
- Concentration: All 26.1.x versions show 0% pass rate; 25.8.x has partial passes
- Error signature: Consistent — ExpectTimeoutError 600s
Assessment: Flaky (pre-existing)
Recommendation: Known unstable test, no action needed for PR verification
Both Swarms failures are confirmed as pre-existing instability by previous verification reports (PR #1575, PR #1583).
Verdict: Ready to merge after audit review — No unresolved PR-related failures.
4 participants
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Different improvement for remote initiator
Documentation entry for user-facing changes
With remote initiator feature queries like
rewrites as
'remote_host' is a random host from 'swarm' cluster
See #756
Current PR introduces the next improvements:
object_storage_remote_initiatorauth works incorrectly #1570 - uses username and password if access to cluster requires it. Throws exception if cluster uses common secret, this should be solved in future PRs.object_storage_remote_initiatorwith different cluster name #1571 - new settingobject_storage_remote_initiator_clusterallows to chooseremote_hostfrom different cluster, not only fromswarmremotequery did not work with additional setting inside function, likeremote('remote_host', iceberg(..., SETTINGS iceberg_metadata_file_path='path/to/metadata.json')). Now must work correctly.remote('remote_host', icebergCluster('remote_cluster', ...))clusterremote_clustercan be defined only onremote_hostand unknown on current initial host. Removed cluster check on early stage, this allows to execute such queries.CI/CD Options
Exclude tests:
Regression jobs to run: