Remove useless grpc connections (#233)

[ti-chi-bot]

This is an automated cherry-pick of #233

Root cause: illegal connections are not cleaned up when grpc call is failed
Fix: We remove useless connections when grpc call fails with unavailable or cancelled codes
Related issue: pingcap/tiflash#9806

Summary by CodeRabbit

• Bug Fixes

  • Improved handling and propagation of gRPC errors, ensuring failed RPCs mark unhealthy endpoints for removal.
  • More reliable cleanup of unavailable connections during background scans.

• New Features

  • Background health monitoring and automatic removal of bad connections.
  • RPC client now runs as a background task and is stopped earlier during cluster shutdown for more predictable lifecycle behavior.

[ti-chi-bot]

@gengliqi This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5434f039-1741-4af2-89cb-b18bcfd39b27

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gengliqi, wshwsh12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [gengliqi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-05-22 10:34:32.314015011 +0000 UTC m=+2742.284180067: ☑️ agreed by gengliqi.
• 2026-05-22 10:35:46.730759148 +0000 UTC m=+2816.700924205: ☑️ agreed by wshwsh12.

[gengliqi]

/retest