-
Notifications
You must be signed in to change notification settings - Fork 32
Add AWS Process Credential Resolver #658
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jonathan343
wants to merge
14
commits into
develop
Choose a base branch
from
process-cred-resolver
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
1c2db12
Add simple process credentials resolver
jonathan343 dbf978f
Improve process credentials resolver command handling
jonathan343 e3f0d11
fix type checking errors
jonathan343 2ab7cdd
Simplify example creds names
jonathan343 daf40d1
Fix process credentials timezone handling, JSON error wrapping
jonathan343 cb4f02b
Only allow commands as a list of strings
jonathan343 6ee36dd
Update non-zero ecxeption message based on feedback
jonathan343 713c6f1
Integrate with new credential chain
jonathan343 4c95482
Support Windows command parsing for process credentials
jonathan343 8011fca
Simplify process credential timeout configuration
jonathan343 e189b7b
Harden process credential parsing and add aws_account_id fallback
jonathan343 0c48d90
Drop unused import
jonathan343 c7af394
Address test related feedback
jonathan343 8dbac3e
Minor improvements after self review
jonathan343 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
4 changes: 4 additions & 0 deletions
4
...-core/.changes/next-release/smithy-aws-core-feature-9e2d74d0c5724eacbee1b1af6260ab54.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| { | ||
| "type": "feature", | ||
| "description": "Added process credentials support to the default AWS identity chain through the active profile's `credential_process` setting." | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
123 changes: 123 additions & 0 deletions
123
packages/smithy-aws-core/src/smithy_aws_core/identity/chain/providers/process.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| import shlex | ||
| import sys | ||
|
|
||
| from smithy_core.interfaces.identity import Identity | ||
|
|
||
| from ...components import AWSCredentialsIdentity | ||
| from ...process import ProcessCredentialsResolver | ||
| from ..ordering import Standard, StandardProvider | ||
| from ..provider import ChainSetup | ||
|
|
||
| _CREDENTIAL_PROCESS = "credential_process" | ||
| _ACCOUNT_ID = "aws_account_id" | ||
|
|
||
|
|
||
| def _split_process_command( | ||
| command: str, | ||
| *, | ||
| platform: str | None = None, | ||
| ) -> list[str]: | ||
| """Split a process command according to the host platform's quoting rules.""" | ||
| if platform is None: | ||
| platform = sys.platform | ||
| if platform == "win32": | ||
| return _split_windows_command(command) | ||
| return shlex.split(command) | ||
|
|
||
|
|
||
| def _split_windows_command(command: str) -> list[str]: | ||
| """Split a command using botocore's strict form of the Microsoft C runtime rules. | ||
|
|
||
| The underlying runtime rules are documented at: | ||
| https://learn.microsoft.com/en-us/cpp/cpp/main-function-command-line-args#parsing-c-command-line-arguments | ||
| """ | ||
| arguments: list[str] = [] | ||
| argument: list[str] = [] | ||
| argument_started = False | ||
| in_quotes = False | ||
| backslashes = 0 | ||
|
|
||
| for character in command: | ||
| if character == "\\": | ||
| # Delay emitting backslashes until we know whether a quote follows. | ||
| backslashes += 1 | ||
| argument_started = True | ||
| continue | ||
|
|
||
| if character == '"': | ||
| # Pairs become literal backslashes; an odd remainder escapes the quote. | ||
| literal_backslashes, escaped_quote = divmod(backslashes, 2) | ||
| argument.extend("\\" * literal_backslashes) | ||
| backslashes = 0 | ||
| argument_started = True | ||
| if escaped_quote: | ||
| argument.append('"') | ||
| else: | ||
| in_quotes = not in_quotes | ||
| continue | ||
|
|
||
| if backslashes: | ||
| # Without a following quote, backslashes are literal. | ||
| argument.extend("\\" * backslashes) | ||
| backslashes = 0 | ||
|
|
||
| # Only spaces and tabs outside quotes delimit Windows arguments. | ||
| if character in (" ", "\t") and not in_quotes: | ||
| # This preserves empty quoted arguments while ignoring extra whitespace. | ||
| if argument_started: | ||
| arguments.append("".join(argument)) | ||
| argument = [] | ||
| argument_started = False | ||
| continue | ||
|
|
||
| argument.append(character) | ||
| argument_started = True | ||
|
|
||
| if in_quotes: | ||
| raise ValueError(f"No closing quotation in string: {command}") | ||
|
|
||
| if backslashes: | ||
| argument.extend("\\" * backslashes) | ||
| if argument_started: | ||
| arguments.append("".join(argument)) | ||
|
|
||
| return arguments | ||
|
|
||
|
|
||
| class ProfileProcessCredentialsProvider: | ||
| """Adds a process credential resolver configured by the active profile.""" | ||
|
|
||
| @property | ||
| def name(self) -> str: | ||
| """Return the canonical provider name.""" | ||
| return StandardProvider.PROFILE_CREDENTIAL_PROCESS.canonical_name | ||
|
|
||
| @property | ||
| def ordering(self) -> Standard: | ||
| """Return the provider's standard chain position.""" | ||
| return Standard(slot=StandardProvider.PROFILE_CREDENTIAL_PROCESS) | ||
|
|
||
| async def setup(self, identity_type: type[Identity], setup: ChainSetup) -> None: | ||
| """Add a resolver when the active profile configures a credential process.""" | ||
| if identity_type is not AWSCredentialsIdentity: | ||
| return | ||
|
|
||
| config_file = setup.config_file | ||
| profile_name = setup.profile_name | ||
| if config_file is None or profile_name is None: | ||
| return | ||
|
|
||
| command = config_file.get(profile_name, _CREDENTIAL_PROCESS) | ||
| if not command: | ||
| return | ||
|
|
||
| # The process output's AccountId takes precedence; the profile's | ||
| # aws_account_id is only used as a fallback. | ||
| setup.add_terminal_resolver( | ||
| ProcessCredentialsResolver( | ||
| _split_process_command(command), | ||
| account_id=config_file.get(profile_name, _ACCOUNT_ID), | ||
| ) | ||
| ) | ||
141 changes: 141 additions & 0 deletions
141
packages/smithy-aws-core/src/smithy_aws_core/identity/process.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| import asyncio | ||
|
jonathan343 marked this conversation as resolved.
|
||
| import json | ||
| from datetime import UTC, datetime | ||
| from typing import TypeGuard, cast | ||
|
|
||
| from smithy_core.aio.interfaces.identity import IdentityResolver | ||
| from smithy_core.exceptions import SmithyIdentityError | ||
|
|
||
| from .components import AWSCredentialsIdentity, AWSIdentityProperties | ||
|
|
||
|
|
||
| def _is_command_list(command: object) -> TypeGuard[list[str]]: | ||
| if not isinstance(command, list) or not command: | ||
| return False | ||
| return all(isinstance(argument, str) for argument in cast(list[object], command)) | ||
|
|
||
|
|
||
| class ProcessCredentialsResolver( | ||
| IdentityResolver[AWSCredentialsIdentity, AWSIdentityProperties] | ||
| ): | ||
| """Resolves AWS Credentials from a process. | ||
|
|
||
| :param command: The process command and arguments to execute, as a | ||
| non-empty list of strings. | ||
| :param timeout: Maximum time in seconds to wait for the process to complete. | ||
| :param account_id: Fallback account ID to associate with the resolved | ||
| credentials when the process output does not include an ``AccountId``. | ||
| """ | ||
|
|
||
| def __init__( | ||
| self, | ||
| command: list[str], | ||
|
jonathan343 marked this conversation as resolved.
|
||
| *, | ||
| timeout: float | None = None, | ||
| account_id: str | None = None, | ||
| ) -> None: | ||
| if not _is_command_list(command): | ||
| raise ValueError("command must be a non-empty list of strings") | ||
| self._command = list(command) | ||
| self._timeout = timeout | ||
| self._account_id = account_id | ||
| self._credentials: AWSCredentialsIdentity | None = None | ||
|
|
||
| async def get_identity( | ||
| self, *, properties: AWSIdentityProperties | ||
| ) -> AWSCredentialsIdentity: | ||
| if self._credentials is not None: | ||
| # Long-term credentials (no expiration) should always be reused | ||
| if self._credentials.expiration is None: | ||
| return self._credentials | ||
| # Temporary credentials should be reused if not expired | ||
| if datetime.now(UTC) < self._credentials.expiration: | ||
|
jonathan343 marked this conversation as resolved.
|
||
| return self._credentials | ||
|
|
||
| try: | ||
| process = await asyncio.create_subprocess_exec( | ||
| *self._command, | ||
| stdout=asyncio.subprocess.PIPE, | ||
| stderr=asyncio.subprocess.PIPE, | ||
| ) | ||
| except OSError as e: | ||
| raise SmithyIdentityError(f"Credential process failed to start: {e}") from e | ||
|
|
||
| try: | ||
| stdout, stderr = await asyncio.wait_for( | ||
| process.communicate(), timeout=self._timeout | ||
| ) | ||
| except TimeoutError as e: | ||
| if process.returncode is None: | ||
| try: | ||
| process.kill() | ||
| except ProcessLookupError: | ||
| pass | ||
| await process.wait() | ||
| raise SmithyIdentityError( | ||
| f"Credential process timed out after {self._timeout} seconds" | ||
| ) from e | ||
|
|
||
| if process.returncode != 0: | ||
| raise SmithyIdentityError( | ||
| f"Credential process failed with exit code {process.returncode}: " | ||
| f"{stderr.decode('utf-8', errors='replace')}" | ||
| ) | ||
| # These exceptions retain the full process output, which may contain | ||
| # credentials. Suppress chaining to avoid exposing it in tracebacks. | ||
| try: | ||
| decoded = stdout.decode("utf-8") | ||
| creds = json.loads(decoded) | ||
| except UnicodeDecodeError as e: | ||
| raise SmithyIdentityError( | ||
| "Credential process output is not valid UTF-8 " | ||
| f"at byte {e.start}: {e.reason}" | ||
| ) from None | ||
| except json.JSONDecodeError as e: | ||
| raise SmithyIdentityError( | ||
| "Credential process output is not valid JSON " | ||
| f"at line {e.lineno}, column {e.colno}: {e.msg}" | ||
| ) from None | ||
|
|
||
| version = creds.get("Version") | ||
| if version != 1: | ||
| raise SmithyIdentityError( | ||
| f"Unsupported version '{version}' for credential process provider, supported versions: 1" | ||
| ) | ||
| access_key_id = creds.get("AccessKeyId") | ||
| secret_access_key = creds.get("SecretAccessKey") | ||
| session_token = creds.get("SessionToken") | ||
| expiration = creds.get("Expiration") | ||
| # Prefer the process output's AccountId, falling back to the profile's | ||
| # aws_account_id when the process omits it. | ||
| account_id = creds.get("AccountId") or self._account_id | ||
|
|
||
| if expiration is not None: | ||
| try: | ||
| dt = datetime.fromisoformat(expiration) | ||
| except (TypeError, ValueError) as e: | ||
| raise SmithyIdentityError( | ||
| "Invalid credential process Expiration; " | ||
| f"expected an ISO 8601 string: {e}" | ||
| ) from e | ||
| expiration = dt.astimezone(UTC) if dt.tzinfo else dt.replace(tzinfo=UTC) | ||
|
|
||
| if access_key_id is None or secret_access_key is None: | ||
| raise SmithyIdentityError( | ||
| "AccessKeyId and SecretAccessKey are required for process credentials" | ||
| ) | ||
|
|
||
| self._credentials = AWSCredentialsIdentity( | ||
| access_key_id=access_key_id, | ||
| secret_access_key=secret_access_key, | ||
| session_token=session_token, | ||
| expiration=expiration, | ||
| account_id=account_id, | ||
| ) | ||
| return self._credentials | ||
|
|
||
| async def invalidate(self) -> None: | ||
| """Discard cached credentials so the next resolution reruns the process.""" | ||
| self._credentials = None | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note for reviewer: This is inspired by botocore's _windows_shell_split utility function.