修复:避免多 tunnel 实例心跳导致代理节点事件重复入库#673
Open
zbsdsb wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
现象
proxy_node_events会在短时间内写入大量重复的tunnel_err记录,事件表持续膨胀。实际数据中,同一个时间点可以出现十几万条记录,但
detail和event_metadata只有少量不同值;同时,节点上报的累计error_events_total远小于数据库中的事件行数。这说明同一批近期 tunnel 错误被心跳重复持久化,并不是真实发生了数百万次独立错误。代理节点仍然可以显示在线,因为问题发生在心跳指标和错误事件的累计逻辑,不代表 tunnel 连接一定离线。
触发条件
核心触发条件是:多个
aether-tunnel进程共用同一个代理节点 ID。例如同一台机器同时通过 systemd 和 Docker 启动 tunnel:
node_id交替发送心跳。recent_tunnel_errors根据这个错误的 delta 被重复写入proxy_node_events。相同 heartbeat 重试、heartbeat ID 回退或乱序,也可能造成相同事件被重复处理。
根因
新版 tunnel 客户端已经在心跳顶层发送:
heartbeat_session_id:标识当前 tunnel 进程/心跳任务会话。heartbeat_id:同一 session 内严格递增,重试时复用原 ID。但 gateway 的
InternalTunnelHeartbeatRequest没有读取heartbeat_session_id,Serde 会把该字段当作未知字段丢弃。数据层因此只能看到累计计数器,无法判断两次心跳是否来自同一个 tunnel 进程,也无法可靠识别 session 切换、重放和乱序。修复方案
本 PR 直接复用 tunnel 已经发送的 heartbeat cursor,不新增数据库字段,也不需要数据库迁移:
heartbeat_session_id。heartbeat_session_id和heartbeat_id写入代理节点 metadata。heartbeat_id严格大于上一次时,才计算累计指标 delta。recent_tunnel_errors:这样即使两个 tunnel 实例使用同一个节点 ID 交替上报,也不会再把各自的累计计数器反复计算成新增事件。
验证
已覆盖以下场景:
本地验证结果:
gateway 的 HTTP 回归测试已随 PR 提交,由仓库 Linux CI 继续执行。