Skip to content

bug: UNAVAILABLE after rolling restart #261

@asmyasnikov

Description

@asmyasnikov
[2026-05-04 20:56:38]: INFO: YDBFv2: [execYdbFunc]: Retry attempt 1/3
[2026-05-04 20:56:38]: INFO: YDBFv2: [connectYDB]: Using PROJECT_ID: 1
[2026-05-04 20:56:38]: INFO: YDBFv2: [connectYDB]: Connecting to YDB endpoint: lb.HIDDEN.ydb.mdb.yandexcloud.net:2135
[2026-05-04 20:56:38]: INFO: YDBFv2: [connectYDB]: DNS resolved [HIDDEN.ydb.mdb.yandexcloud.net] to IPs: 81.26.185.44, 93.77.182.188
[2026-05-04 20:56:38]: INFO: YDBFv2: [connectYDB]: Creating FIRST YDB connection for project 1
[2026-05-04 20:56:40]: ERR: YDBFv2: [connectYDB]: Error creating YDB client: YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused
[2026-05-04 20:56:40]: ERR: YDBFv2: [execYdbFunc]: Exception caught: YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused
[2026-05-04 20:56:40]: ERR: YDBFv2: GRPC_UNAVAILABLE - YDB service temporarily down
[2026-05-04 20:56:40]: CRITICAL: YDBFv2: GRPC error on endpoint: unknown
[2026-05-04 20:56:40]: CRITICAL: YDBFv2: Failed YDB node IP: 81.26.185.44:2135
[2026-05-04 20:56:42]: INFO: YDBFv2: [execYdbFunc]: Retry attempt 2/3
[2026-05-04 20:56:42]: INFO: YDBFv2: [connectYDB]: Using PROJECT_ID: 1
[2026-05-04 20:56:42]: INFO: YDBFv2: [connectYDB]: Connecting to YDB endpoint: lb.HIDDEN.ydb.mdb.yandexcloud.net:2135
[2026-05-04 20:56:42]: INFO: YDBFv2: [connectYDB]: DNS resolved [lb.HIDDEN.ydb.mdb.yandexcloud.net] to IPs: 93.77.182.188, 81.26.185.44
[2026-05-04 20:56:42]: INFO: YDBFv2: [connectYDB]: Creating FIRST YDB connection for project 1
[2026-05-04 20:56:44]: ERR: YDBFv2: [connectYDB]: Error creating YDB client: YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused
[2026-05-04 20:56:44]: ERR: YDBFv2: [execYdbFunc]: Exception caught: YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused
[2026-05-04 20:56:44]: ERR: YDBFv2: GRPC_UNAVAILABLE - YDB service temporarily down
[2026-05-04 20:56:44]: CRITICAL: YDBFv2: GRPC error on endpoint: unknown
[2026-05-04 20:56:44]: CRITICAL: YDBFv2: Failed YDB node IP: 81.26.185.44:2135
[2026-05-04 20:56:46]: INFO: YDBFv2: [execYdbFunc]: Retry attempt 3/3
[2026-05-04 20:56:46]: INFO: YDBFv2: [getYdbSession]: getYdbSession_ydb_retry_m5 > 2 and = 3
[2026-05-04 20:56:46]: INFO: YDBFv2: [getYDB]: newone true 
[2026-05-04 20:56:46]: INFO: YDBFv2: [connectYDB]: Using PROJECT_ID: 1
[2026-05-04 20:56:46]: INFO: YDBFv2: [connectYDB]: Connecting to YDB endpoint: lb.HIDDEN.ydb.mdb.yandexcloud.net:2135
[2026-05-04 20:56:46]: INFO: YDBFv2: [connectYDB]: DNS resolved [lb.HIDDEN.ydb.mdb.yandexcloud.net] to IPs: 81.26.185.44, 93.77.182.188
[2026-05-04 20:56:46]: INFO: YDBFv2: [connectYDB]: Creating NEW YDB connection for project 1
[2026-05-04 20:56:48]: ERR: YDBFv2: [connectYDB]: Error creating YDB client: YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused
[2026-05-04 20:56:48]: ERR: YDBFv2: [execYdbFunc]: Exception caught: YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused
[2026-05-04 20:56:48]: ERR: YDBFv2: GRPC_UNAVAILABLE - YDB service temporarily down
[2026-05-04 20:56:48]: CRITICAL: YDBFv2: GRPC error on endpoint: unknown
[2026-05-04 20:56:48]: CRITICAL: YDBFv2: Failed YDB node IP: 81.26.185.44:2135
[2026-05-04 20:56:50]: INFO: YDBFv2: [select_ydb]: select_result = {"status":"error","code":"106","msg":"error with ydb","moreinfo":"YDB Discovery ListEndpoints (status code GRPC_UNAVAILABLE 14): failed to connect to all addresses; last error: UNKNOWN: ipv4:81.26.185.44:2135: Failed to connect to remote host: Connection refused","conn":"already conn (age: 300s)","data":""}

Гипотезы:

  1. на клиенте закэшировались ответы DNS.
    После обновления YDB IP-адреса узлов YDB поменялись и клиент не смог достучаться по старым IP-адресам.
    Возможный фикс: принудительно сбрасывать кэш DNS (force resolve fqdn)
  2. IPv4 адреса узлов YDB недоступны клиентам
    Возможный фикс 1: убрать IPv4 адреса из записей FQDN узлов YDB
    Возможный фикс 2: фильтровать IPv4 адреса из записей FQDN узлов YDB (ходить только в IPv6)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions