Skip to content

az extension add --name azure-devops reaches out to pypi.org and flagged in build pipeline #29109

Description

@feiyushi

Describe the bug

Due to internal security push related to OSS network isolation, endpoint such as pypi.org is flagged during the build. The flagged task uses az extension add --name azure-devops which reaches out to https://pypi.org/simple/distro/ to search for versions of distro.

Related command

az extension add --name azure-devops

Errors

1 location(s) to search for versions of distro:
*https://pypi.org/simple/distro/
Fetching project page and analyzing links: https://pypi.org/simple/distro/
Getting page https://pypi.org/simple/distro/
Found index url https://pypi.org/simple/
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/distro/ HTTP/1.1" 200 4086

Issue script & Debug output

cli.knack.cli: Command arguments: ['extension', 'add', '--name', 'azure-devops', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f636dd90040>, <function OutputProducer.on_global_arguments at 0x7f636dd36200>, <function CLIQuery.on_global_arguments at 0x7f636db4fce0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'extension': ['azure.cli.command_modules.extension']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: extension 0.001 1 7
cli.azure.cli.core: Total (1) 0.001 1 7
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: Loaded 1 groups, 7 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : extension add
cli.azure.cli.core: Command table: extension add
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f636cde4e00>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-06-05.22-14-41.extension_add.2379.log'.
az_command_data_logger: command args: extension add --name {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f636ce39ee0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f636ce51120>, <function register_cache_arguments..add_cache_arguments at 0x7f636ce51260>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f636dd362a0>, <function CLIQuery.handle_query_parameter at 0x7f636db4fd80>, <function register_ids_argument..parse_ids_arguments at 0x7f636ce511c0>]
cli.azure.cli.core.extension.operations: Default enabled including preview versions for extension installation now. Disabled in future release. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only.
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
urllib3.connectionpool: https://aka.ms:443 "GET /azure-cli-extension-index-v1 HTTP/1.1" 301 0
urllib3.connectionpool: Starting new HTTPS connection (1): azcliextensionsync.blob.core.windows.net:443
urllib3.connectionpool: https://azcliextensionsync.blob.core.windows.net:443 "GET /index1/index.json HTTP/1.1" 200 4128036
cli.azure.cli.core.extension._resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension._resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension._resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension.resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension.resolve: Chosen {'downloadUrl': 'https://github.com/Azure/azure-devops-cli-extension/releases/download/20240514.1/azure_devops-1.0.1-py2.py3-none-any.whl', 'filename': 'azure_devops-1.0.1-py2.py3-none-any.whl', 'metadata': {'azext.minCliCoreVersion': '2.30.0', 'classifiers': ['Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'License :: OSI Approved :: MIT License'], 'extensions': {'python.details': {'contacts': [{'email': 'VSTS_Social@microsoft.com', 'name': 'Microsoft', 'role': 'author'}], 'document_names': {'description': 'DESCRIPTION.rst'}, 'project_urls': {'Home': 'https://github.com/Microsoft/azure-devops-cli-extension'}}}, 'extras': [], 'generator': 'bdist_wheel (0.30.0)', 'license': 'MIT', 'metadata_version': '2.0', 'name': 'azure-devops', 'run_requires': [{'requires': ['distro (==1.3.0)', 'distro==1.3.0']}], 'summary': 'Tools for managing Azure DevOps.', 'version': '1.0.1'}, 'sha256Digest': 'f300d0288f017148514ebe6f5912aef10c7a6f29bdc0c916b922edf1d75bc7db'}
cli.azure.cli.core.extension.operations: Extension source is url? True
cli.azure.cli.core.extension.operations: Downloading https://github.com/Azure/azure-devops-cli-extension/releases/download/20240514.1/azure_devops-1.0.1-py2.py3-none-any.whl to /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
urllib3.connectionpool: Starting new HTTPS connection (1): github.com:443
urllib3.connectionpool: https://github.com:443 "GET /Azure/azure-devops-cli-extension/releases/download/20240514.1/azure_devops-1.0.1-py2.py3-none-any.whl HTTP/1.1" 302 0
urllib3.connectionpool: Starting new HTTPS connection (1): objects.githubusercontent.com:443
urllib3.connectionpool: https://objects.githubusercontent.com:443 "GET /github-production-release-asset-2e65be/107708057/77ec1ba4-6f10-4d2b-bb30-9c1d420c4fbc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240605%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240605T221442Z&X-Amz-Expires=300&X-Amz-Signature=61bc135cef10bed4ce7fe1e02db2ce85027decb9161dfdff54da09594b3bdad0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107708057&response-content-disposition=attachment%3B%20filename%3Dazure_devops-1.0.1-py2.py3-none-any.whl&response-content-type=application%2Foctet-stream HTTP/1.1" 200 1195727
cli.azure.cli.core.extension.operations: Downloaded to /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core.extension.operations: Validating the extension /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core.extension.operations: Checksum of /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl is OK
cli.azure.cli.core.extension.operations: Validation successful on /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core.extension.operations: Linux distro check: Reading from: /etc/apt/sources.list.d/azure-cli.list
cli.azure.cli.core.extension.operations: Linux distro check: An error occurred while checking linux distribution version source list consistency.
cli.azure.cli.core.extension.operations: [Errno 2] No such file or directory: '/etc/apt/sources.list.d/azure-cli.list'
cli.azure.cli.core.extension.operations: Executing pip with args: ['install', '--target', '/root/.azure/cliextensions/azure-devops', '/tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension.operations: Running: ['/opt/az/bin/python3', '-m', 'pip', 'install', '--target', '/root/.azure/cliextensions/azure-devops', '/tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']
cli.azure.cli.core.extension.operations: Using pip 24.0 from /opt/az/lib/python3.11/site-packages/pip (python 3.11)
Non-user install due to --prefix or --target option
Created temporary directory: /tmp/pip-target-veukpuvo
Created temporary directory: /tmp/pip-build-tracker-60z43nj2
Initialized build tracking at /tmp/pip-build-tracker-60z43nj2
Created build tracker: /tmp/pip-build-tracker-60z43nj2
Entered build tracker: /tmp/pip-build-tracker-60z43nj2
Created temporary directory: /tmp/pip-install-xdcqadi6
Created temporary directory: /tmp/pip-ephem-wheel-cache-t6ychn8h
Processing /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
1 location(s) to search for versions of distro:
https://pypi.org/simple/distro/
Fetching project page and analyzing links: https://pypi.org/simple/distro/
Getting page https://pypi.org/simple/distro/
Found index url https://pypi.org/simple/

Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/distro/ HTTP/1.1" 200 4086
Fetched page https://pypi.org/simple/distro/ as application/vnd.pypi.simple.v1+json
Found link https://files.pythonhosted.org/packages/b7/ff/876ab097c769295f880c9056d09c934f5c7c4c6054df1a83953b73f85f73/distro-0.5.0.tar.gz (from https://pypi.org/simple/distro/), version: 0.5.0
Found link https://files.pythonhosted.org/packages/01/9c/ea1b152ac247a5747598168b88ae82eb742461e2e556262ae741e69bd30f/distro-0.6.0.tar.gz (from https://pypi.org/simple/distro/), version: 0.6.0
Found link https://files.pythonhosted.org/packages/15/97/e2d5863d03cd01b250e51117be031dab2ec0916efc5915f5094f5fd7602c/distro-1.0.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.0
Found link https://files.pythonhosted.org/packages/80/4e/22225a92917ebf7780ddb972e14add3d710c718ac00f152c7811ca07b4f3/distro-1.0.1-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.1
Found link https://files.pythonhosted.org/packages/0d/c5/de784640f0b434799d6eecb63baea9c099e7fe6c0908b4036c3a5200c281/distro-1.0.1.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.1
Found link https://files.pythonhosted.org/packages/64/4d/2d450e0a6dcf394bc0c581e499a66e00d255938f74db6b15748373d36cc4/distro-1.0.2-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.2
Found link https://files.pythonhosted.org/packages/42/ac/89b295d2784d450ca71ac6f3665cb90f07afe0928e4436af627983faf2b1/distro-1.0.2.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.2
Found link https://files.pythonhosted.org/packages/4a/c2/50dbfaac03c14b8e155e4329553ef1f73a51a5c3cece8fb62e849c0785ed/distro-1.0.3-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.3
Found link https://files.pythonhosted.org/packages/df/65/a8a94c1e069cf5f5bffab2a6b88d5a78089c60f2916e96d18b625731f191/distro-1.0.3.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.3
Found link https://files.pythonhosted.org/packages/b5/82/363544dcfa3e7f1478e6839aa929a95ac6e1b0c3b56a277e6a6ef3ace2c9/distro-1.0.4-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.4
Found link https://files.pythonhosted.org/packages/aa/4e/2cf3e7f67abe101c053af838f8d9b3b5911fd9360b498a1ba66a23d1ed46/distro-1.0.4.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.4
Found link https://files.pythonhosted.org/packages/b0/55/29bfd4d4d4149e860ed01aa446108eb17b240997b746c06a2d0c8ce04f69/distro-1.1.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.1.0
Found link https://files.pythonhosted.org/packages/21/7b/14198029b49abdf80c6b8aadd9862f863b683dc4d3c2418f01bc6fad9fa3/distro-1.1.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.1.0
Found link https://files.pythonhosted.org/packages/c1/e4/933159b5f7f9f5b7ae463e76f58da84a30d0943ab3c162c366a9ad95f01e/distro-1.2.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.2.0
Found link https://files.pythonhosted.org/packages/b2/2e/e4b8b7f947465474e58bc9dbaa6ea8c4b4cc9e845711c0fc2f66601e464b/distro-1.2.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.2.0
Found link https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.3.0
Found link https://files.pythonhosted.org/packages/d2/42/3b059929a920cd9d4e91e7a5e35f0d2ed75211f8f4e877be9d1bde9fdf46/distro-1.3.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.3.0
Found link https://files.pythonhosted.org/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.4.0
Found link https://files.pythonhosted.org/packages/ca/e3/78443d739d7efeea86cbbe0216511d29b2f5ca8dbf51a6f2898432738987/distro-1.4.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.4.0
Found link https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.5.0
Found link https://files.pythonhosted.org/packages/a6/a4/75064c334d8ae433445a20816b788700db1651f21bdb0af33db2aab142fe/distro-1.5.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.5.0
Found link https://files.pythonhosted.org/packages/b3/8d/a0a5c389d76f90c766e956515d34c3408a1e18f60fbaa08221d1f6b87490/distro-1.6.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.6.0
Found link https://files.pythonhosted.org/packages/a5/26/256fa167fe1bf8b97130b4609464be20331af8a3af190fb636a8a7efd7a2/distro-1.6.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.6.0
Found link https://files.pythonhosted.org/packages/e1/54/d08d1ad53788515392bec14d2d6e8c410bffdc127780a9a4aa8e6854d502/distro-1.7.0-py3-none-any.whl (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.7.0
Found link https://files.pythonhosted.org/packages/b5/7e/ddfbd640ac9a82e60718558a3de7d5988a7d4648385cf00318f60a8b073a/distro-1.7.0.tar.gz (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.7.0
Found link https://files.pythonhosted.org/packages/f4/2c/c90a3adaf0ddb70afe193f5ebfb539612af57cffe677c3126be533df3098/distro-1.8.0-py3-none-any.whl (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.8.0
Found link https://files.pythonhosted.org/packages/4b/89/eaa3a3587ebf8bed93e45aa79be8c2af77d50790d15b53f6dfc85b57f398/distro-1.8.0.tar.gz (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.8.0
Found link https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.9.0
Found link https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.9.0
Skipping link: not a file: https://pypi.org/simple/distro/
Given no hashes to check 2 links for project 'distro': discarding no candidates
Collecting distro==1.3.0 (from azure-devops==1.0.1)
Obtaining dependency information for distro==1.3.0 from https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl.metadata
Created temporary directory: /tmp/pip-unpack-8al0l3s1
Starting new HTTPS connection (1): files.pythonhosted.org:443
https://files.pythonhosted.org:443 "GET /packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl.metadata HTTP/1.1" 200 1362
Downloading distro-1.3.0-py2.py3-none-any.whl.metadata (1.4 kB)
Created temporary directory: /tmp/pip-metadata-39x6pldp
Created temporary directory: /tmp/pip-unpack-jervwsa

https://files.pythonhosted.org:443 "GET /packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl HTTP/1.1" 200 16807
Downloading distro-1.3.0-py2.py3-none-any.whl (16 kB)
Downloading link https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/) to /tmp/pip-unpack-jervwsa
/distro-1.3.0-py2.py3-none-any.whl
Installing collected packages: distro, azure-devops

Creating /tmp/pip-target-veukpuvo/bin
changing mode of /tmp/pip-target-veukpuvo/bin/distro to 755

Successfully installed azure-devops-1.0.1 distro-1.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removed build tracker: '/tmp/pip-build-tracker-60z43nj2'

cli.azure.cli.core.extension.operations: Saved the whl to /root/.azure/cliextensions/azure-devops/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core: Command index has been invalidated.
cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f636ce3a340>, <function _x509_from_base64_to_hex_transform at 0x7f636ce3a3e0>]
cli.knack.cli: Event: CommandInvoker.OnFilterResult []
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f636cde5080>]
az_command_data_logger: exit code: 0
cli.main: Command ran in 3.023 seconds (init: 0.099, invoke: 2.924)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3548 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/init.py /root/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

no internet connection to external endpoints.
OSS netiso doc: https://eng.ms/docs/cloud-ai-platform/azure-edge-platform-aep/aep-engineering-systems/productivity-and-experiences/network-isolation/oss/wave1_oss
looks like not all extension install reaches out to pypi.org. for example aks-preview extension add doesn't connect to pypi.org.

Environment Summary

azure-cli 2.61.0

core 2.61.0
telemetry 1.1.0

Extensions:
aks-preview 4.0.0b5
azure-devops 1.0.1

Dependencies:
msal 1.28.0
azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, May 16 2024, 03:50:11) [GCC 10.2.1 20210110]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    DevOpsService AttentionThis issue is responsible by Azure service team.act-platform-engineering-squadbugThis issue requires a change to an existing behavior in the product in order to be resolved.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions