-
Notifications
You must be signed in to change notification settings - Fork 148
If the token is null, the connection hangs (#458) #876
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
3cb80da
11579f5
f255f4a
3afd559
7e3e520
8aa4e7d
9ea4bb8
121bd0b
ac27f93
2d6f729
eb1af56
bf48dae
ead14dd
91e698e
fbdb5dc
c10bee7
dcba798
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| import base64 | ||
| import errno | ||
| import hashlib | ||
| import json | ||
| import logging | ||
|
|
@@ -62,18 +63,43 @@ def refresh(self) -> Token: | |
|
|
||
|
|
||
| class OAuthManager: | ||
| # Default maximum time (in seconds) to wait for the browser OAuth redirect | ||
| # callback before giving up. Without a bound, the local callback server | ||
| # would block forever in a headless environment (e.g. a notebook/job with | ||
| # no browser), making the connection appear to hang indefinitely. See issue | ||
| # #458. This is generous for interactive logins (slow MFA, IdP re-auth, SSO | ||
| # redirects) and is a fixed ceiling for connector end users: there is no | ||
| # public ``connect()``/``Connection`` parameter to change it. The | ||
| # ``redirect_callback_timeout_seconds`` argument below is an internal-only | ||
| # override for callers constructing ``OAuthManager`` directly (private API); | ||
| # it is not plumbed through ``DatabricksOAuthProvider`` or the public | ||
|
peco-review-bot[bot] marked this conversation as resolved.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The default
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. I made the fail-fast change the reviewer suggested (via the reliable Pushed bf48dae (bundled with 1 other thread(s)). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for taking this on. I re-checked the head revision and the fail-fast on
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The fail-fast is already present on the current head. In src/databricks/sql/auth/oauth.py, __get_authorization_code has an There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I re-read the full
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The requested fail-fast is already present on the current head and no code change is warranted. src/databricks/sql/auth/oauth.py __get_authorization_code has an |
||
| # connection kwargs. It accepts a ``float`` so callers (and tests) may pass | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The 5-minute ceiling is a deliberate and documented behavior change, but note it is a hard cap for interactive U2M logins with no public escape hatch: the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Reviewer is calling out an intentional, already-documented tradeoff (the 5-minute REDIRECT_CALLBACK_TIMEOUT_SECONDS hard cap with no public escape hatch), not requesting a change in this PR. The rationale, the fixed ceiling, and the internal-only redirect_callback_timeout_seconds override are all documented in the oauth.py:63-83 comment block the reviewer is reading. Their only suggestion is conditional/future ("consider surfacing it as a connection kwarg IF slow-login reports appear"), which would be a public-API change — out of scope for this bug-fix PR and belongs in a separate PR / product decision. Flagging for a human to decide whether to add a public connect() timeout kwarg later; nothing to action in this diff. |
||
| # fractional-second timeouts; this class default is a whole number of | ||
| # seconds. | ||
| REDIRECT_CALLBACK_TIMEOUT_SECONDS = 60 * 5 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — This introduces a hard 5-minute cap on every U2M interactive login, not just the headless failure case, and — as the docstring itself notes — there is no public Separately, for the exact reported scenario in #458 (headless notebook/job), the common path is
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Agreed the concern is valid, but neither remedy is actionable in this bug-fix PR. (a) Exposing the redirect-callback timeout as a public/documented knob is a public API change — it must be plumbed through DatabricksOAuthProvider and the public connect()/Connection kwargs (the code deliberately keeps it internal-only), which is out of scope here and belongs in a separate API-review PR. (b) Fast-failing when webbrowser.open_new() returns False (the common #458 headless path) conflicts with a deliberate, documented decision (oauth.py ~L222-237) that a falsy return is not a reliable cross-platform headless signal and would break working interactive logins. The remaining question — the correct default ceiling and whether to expose a public override for a widely-consumed connector, given that logins >5 min that previously succeeded will now fail — is a product/API judgment call for a human maintainer, and further bot back-and-forth won't resolve it. Flagging for human review. |
||
|
|
||
| def __init__( | ||
| self, | ||
| port_range: List[int], | ||
| client_id: str, | ||
| idp_endpoint: OAuthEndpointCollection, | ||
| http_client, | ||
| redirect_callback_timeout_seconds: Optional[float] = None, | ||
| ): | ||
| self.port_range = port_range | ||
| self.client_id = client_id | ||
| self.redirect_port = None | ||
| self.idp_endpoint = idp_endpoint | ||
| self.http_client = http_client | ||
| # Fall back to the class default when not overridden. Kept as a | ||
| # lowercase instance attribute (distinct from the ALL_CAPS class | ||
| # constant that provides the default) so the rest of the flow reads a | ||
| # single, clearly per-instance source of truth. | ||
| self._redirect_callback_timeout_seconds = ( | ||
| redirect_callback_timeout_seconds | ||
| if redirect_callback_timeout_seconds is not None | ||
| else self.REDIRECT_CALLBACK_TIMEOUT_SECONDS | ||
| ) | ||
|
|
||
| @staticmethod | ||
| def __token_urlsafe(nbytes=32): | ||
|
|
@@ -128,11 +154,40 @@ def __get_challenge(): | |
|
|
||
| def __get_authorization_code(self, client, auth_url, scope, state, challenge): | ||
| handler = OAuthHttpSingleRequestHandler("Databricks Sql Connector") | ||
| # Bound the read of the accepted connection too (not just the accept | ||
| # wait below). StreamRequestHandler.setup() applies this to the accepted | ||
| # socket via settimeout(), so a client that connects but never completes | ||
| # the request line can no longer block indefinitely. | ||
| handler.timeout = self._redirect_callback_timeout_seconds | ||
|
|
||
| last_error = None | ||
| callback_timed_out = False | ||
| for port in self.port_range: | ||
| try: | ||
| with HTTPServer(("", port), handler) as httpd: | ||
| # Bound how long we wait for the browser redirect callback so | ||
| # that a headless environment (no browser to complete the | ||
| # flow) fails with a clear error instead of hanging forever. | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| # NOTE: httpd.timeout bounds only the wait to ACCEPT an | ||
| # incoming connection (it is the select() timeout used by | ||
| # handle_request). The subsequent read of the HTTP request | ||
| # line is bounded separately by the handler.timeout set | ||
| # above (applied to the accepted socket by | ||
| # StreamRequestHandler.setup()), so a client that connects | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| # but never completes the request can no longer block. The | ||
| # headless "no connection ever arrives" case (issue #458) is | ||
| # covered by this accept-wait timeout. | ||
| httpd.timeout = self._redirect_callback_timeout_seconds | ||
|
|
||
| # HTTPServer.handle_request() returns normally (via | ||
| # handle_timeout()) when the wait elapses without a | ||
| # connection, so record that case to distinguish it from a | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| # received-but-empty callback below. | ||
| def _on_timeout(): | ||
| nonlocal callback_timed_out | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The accept-wait timeout ( 5 minutes is generous and the tradeoff is documented in the class comment, so this is likely acceptable — but note there is no public
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Valid concern, but not actionable in this bug-fix PR — needs a human/maintainer decision. The tradeoff is already documented in the OAuthManager class docstring (oauth.py:63–76): the 5-min ceiling is deliberate and |
||
| callback_timed_out = True | ||
|
|
||
| httpd.handle_timeout = _on_timeout | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| redirect_url = OAuthManager.__get_redirect_url(port) | ||
| auth_req_uri, _, _ = client.prepare_authorization_request( | ||
| authorization_url=auth_url, | ||
|
|
@@ -152,19 +207,80 @@ def __get_authorization_code(self, client, auth_url, scope, state, challenge): | |
| self.redirect_port = port | ||
| break | ||
| except OSError as e: | ||
| if e.errno == 48: | ||
| # Record every bind-time OSError so the fall-through | ||
| # `raise last_error` below always re-raises a real exception. | ||
| # If we only recorded the port-in-use case, an unexpected error | ||
| # on every port (e.g. EACCES, EADDRNOTAVAIL) would leave | ||
| # last_error == None and turn `raise last_error` into | ||
| # `raise None`, masking the real failure with a confusing | ||
| # TypeError. | ||
| last_error = e | ||
| # errno.EADDRINUSE resolves to the platform's value (48 on | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| # macOS, 98 on Linux), so a port-in-use bind failure is | ||
| # recognized cross-platform and downgraded to an info log while | ||
| # we try the next port. | ||
| if e.errno == errno.EADDRINUSE: | ||
| logger.info(f"Port {port} is in use") | ||
| last_error = e | ||
| else: | ||
| logger.warning(f"Unexpected error binding port {port}: {e}") | ||
| except webbrowser.Error as e: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The The new comment on this branch explicitly reasons about avoiding (Anchored to the nearest changed line — see the description for the exact location.)
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Resolved with a code change. The reviewer's concern was valid: I moved Pushed 91e698e. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the follow-up, but the current code at head still has
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Already satisfied at HEAD (working tree clean, up to date with origin/ai/issue-458). In src/databricks/sql/auth/oauth.py the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The re-fetched code at this location still shows
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The requested fix is already present at HEAD and no further code change is warranted, but the review-bot keeps re-flagging it from a stale read, so this needs a human to confirm. In src/databricks/sql/auth/oauth.py the |
||
| # No browser could be launched at all (webbrowser.get() found no | ||
| # runnable browser). This is a strong, reliable headless signal | ||
| # (e.g. a notebook/job with no display), so fail fast with a | ||
| # clear error instead of waiting out the full redirect-callback | ||
| # timeout. We do NOT fail fast merely on a falsy open_new return | ||
| # value: that is unreliable across platforms (it can be falsy | ||
| # even when a browser did open) and would break working | ||
| # interactive logins. Note this fast-fail only covers the case | ||
| # where webbrowser.get() raises; the more common headless case | ||
| # (no browser registered, so open_new() returns False without | ||
| # raising) is NOT caught here and instead falls through to the | ||
| # bounded redirect-callback timeout below, which is the real | ||
| # safeguard against blocking forever. See issue #458. Placed | ||
| # before the generic | ||
| # ``except Exception`` so this RuntimeError propagates instead of | ||
| # being swallowed and retried against the remaining ports. | ||
| msg = ( | ||
| "Could not launch a web browser to complete the U2M OAuth " | ||
| f"login flow ({e}). This looks like a headless environment " | ||
| "(e.g. a notebook or job with no browser), where browser-" | ||
| "based OAuth cannot complete. See issue #458." | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| ) | ||
|
peco-review-bot[bot] marked this conversation as resolved.
|
||
| logger.error(msg) | ||
| raise RuntimeError(msg) from e | ||
| except Exception as e: | ||
| # A non-OSError/non-webbrowser.Error raised here is not | ||
| # port-related (e.g. an OAuth2Error from | ||
| # prepare_authorization_request, or an unexpected programming | ||
| # bug). Binding the same call to a different local port cannot | ||
| # help, so retrying it against every remaining port would only | ||
| # multiply latency and log noise and then re-raise the *last* | ||
| # attempt's error, obscuring the original failure. Fail fast | ||
| # instead, preserving the original exception and traceback via a | ||
| # bare `raise`. This also sidesteps the `raise None` hazard that | ||
| # the OSError branch guards against, since we never fall through | ||
| # to `raise last_error` for this case. | ||
| logger.error("unexpected error: %s", e) | ||
| raise | ||
| if self.redirect_port is None: | ||
| logger.error( | ||
| f"Tried all the ports {self.port_range} for oauth redirect, but can't find free port" | ||
| ) | ||
| raise last_error | ||
|
|
||
| if not handler.request_path: | ||
| msg = f"No path parameters were returned to the callback at {redirect_url}" | ||
| if callback_timed_out: | ||
| msg = ( | ||
| f"Timed out after {self._redirect_callback_timeout_seconds} " | ||
| f"seconds waiting for the OAuth redirect callback at " | ||
| f"{redirect_url}. The login flow was not completed in time — " | ||
| "either the interactive login was not finished within the " | ||
| "timeout, or this is a headless environment (e.g. a notebook " | ||
| "or job with no browser) where no browser can complete the " | ||
| "flow. See issue #458." | ||
| ) | ||
| else: | ||
| msg = f"No path parameters were returned to the callback at {redirect_url}" | ||
| logger.error(msg) | ||
| raise RuntimeError(msg) | ||
| # This is a kludge because the parsing library expects https callbacks | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 Low — This changes an established behavior contract for the interactive U2M flow: previously
handle_request()blocked indefinitely, so a legitimate but slow interactive login (user walks away, slow SSO/MFA/IdP re-auth) would still succeed whenever the browser callback eventually fired. With the new default the whole browser-open→callback window is now capped at 5 minutes, after which a real user who is slow to complete login gets a hardRuntimeErrorrather than eventually connecting. 5 minutes is generous and the tradeoff is deliberate per the comment, so this is informational — but reviewers should confirm 5 min is the intended ceiling for real users, given there is intentionally no public override plumbed throughconnect()/DatabricksOAuthProvider.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Informational (Low) comment — no code change requested. The 5-minute default (REDIRECT_CALLBACK_TIMEOUT_SECONDS = 60 * 5 at oauth.py:77) is a deliberate behavior-contract change for the interactive U2M flow, already documented in-code (issue #458, generous window for slow MFA/SSO/IdP, intentionally no public override via connect()/DatabricksOAuthProvider). The sole open item is a human product decision: confirm 5 min is the intended ceiling for real interactive users and that no public override should be plumbed through connect(). That needs maintainer judgment, not a code edit — flagging for human confirmation.