az storage blob upload --data=@- crashes if input data isn't utf-8

[phire-tc]

Describe the bug

"az storage blob upload" using the --data=@- flag tries to parse its input as utf-8 and crashes if it can't:

$ az storage blob upload --content-type=application/octet-stream --account-name plinfrastructurebackups --account-key {{my-key}} --blob-endpoint {{my-endpoint}} --container-name {{my-container}} --name ipamEXP.custom.xz --data='@-' --overwrite < ~/tmp/ipamEXP.custom.xz
The command failed with an unexpected error. Here is the traceback:
'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Traceback (most recent call last):
  File "/opt/az/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 517, in execute
    args = _pre_command_table_create(self.cli_ctx, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 117, in _pre_command_table_create
    return _expand_file_prefixed_files(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 112, in _expand_file_prefixed_files
    return [_expand_file_prefix(arg) for arg in args]
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 108, in _expand_file_prefix
    return '='.join([arg_split[0], _maybe_load_file(arg_split[1])])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 97, in _maybe_load_file
    return _load_file(poss_file)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 81, in _load_file
    content = sys.stdin.read()
              ^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Related command

az storage blob upload

Errors

The command failed with an unexpected error. Here is the traceback:
'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Traceback (most recent call last):
  File "/opt/az/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 517, in execute
    args = _pre_command_table_create(self.cli_ctx, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 117, in _pre_command_table_create
    return _expand_file_prefixed_files(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 112, in _expand_file_prefixed_files
    return [_expand_file_prefix(arg) for arg in args]
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 108, in _expand_file_prefix
    return '='.join([arg_split[0], _maybe_load_file(arg_split[1])])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 97, in _maybe_load_file
    return _load_file(poss_file)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 81, in _load_file
    content = sys.stdin.read()
              ^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Issue script & Debug output

$ /usr/bin/az storage blob upload --debug --content-type=application/octet-stream --account-name plinfrastructurebackups --account-key {{my-key}} --blob-endpoint {{my-endpoint}} --container-name {{my-container}} --name ipamEXP.custom.xz --data='@-' --overwrite < ~/tmp/ipamEXP.custom.xz
cli.knack.cli: Command arguments: ['storage', 'blob', 'upload', '--debug', '--content-type=application/octet-stream', '--account-name', '{{my=account}}', '--account-key', '{{my-key}}', '--blob-endpoint', '{{my-endpoint}}', '--container-name', '{{my-container}}', '--name', 'ipamEXP.custom.xz', '--data=@-', '--overwrite']
cli.knack.cli: __init__ debug log:
Enable color in terminal.       
cli.knack.cli: Event: Cli.PreExecute []                                                                                
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f19493d77e0>, <function OutputProducer.on_global_arguments at 0x7f19491327a0>, <function CLIQuery.on_global_arguments at 0x7f19491
77ce0>]                       
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/az/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke           
    cmd_result = self.invocation.execute(args)                                                                         
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 517, in execute
    args = _pre_command_table_create(self.cli_ctx, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 117, in _pre_command_table_create
    return _expand_file_prefixed_files(args)            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^            
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 112, in _expand_file_prefixed_files
    return [_expand_file_prefix(arg) for arg in args]
            ^^^^^^^^^^^^^^^^^^^^^^^^        
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 108, in _expand_file_prefix                                                                                                                           
    return '='.join([arg_split[0], _maybe_load_file(arg_split[1])])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 97, in _maybe_load_file    
    return _load_file(poss_file)                                                                                       
           ^^^^^^^^^^^^^^^^^^^^^                                                                                       
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 81, in _load_file      
    content = sys.stdin.read()  
              ^^^^^^^^^^^^^^^^  
  File "<frozen codecs>", line 322, in decode                                                                          
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
                                                           
cli.azure.cli.core.azclierror: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte                                                                                                                                 [2/9010]
Traceback (most recent call last):                                                                                     
  File "/opt/az/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)         
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 517, in execute
    args = _pre_command_table_create(self.cli_ctx, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 117, in _pre_command_table_create
    return _expand_file_prefixed_files(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 112, in _expand_file_prefixed_files
    return [_expand_file_prefix(arg) for arg in args]
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 108, in _expand_file_prefix
    return '='.join([arg_split[0], _maybe_load_file(arg_split[1])])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 97, in _maybe_load_file
    return _load_file(poss_file)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 81, in _load_file
    content = sys.stdin.read()
              ^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
az_command_data_logger: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Traceback (most recent call last):
  File "/opt/az/lib/python3.12/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 517, in execute
    args = _pre_command_table_create(self.cli_ctx, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 117, in _pre_command_table_create
    return _expand_file_prefixed_files(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 112, in _expand_file_prefixed_files
    return [_expand_file_prefix(arg) for arg in args]
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 108, in _expand_file_prefix
    return '='.join([arg_split[0], _maybe_load_file(arg_split[1])])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 97, in _maybe_load_file
    return _load_file(poss_file)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 81, in _load_file
    content = sys.stdin.read()
              ^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f194835f100>]
cli.__main__: Command ran in 0.213 seconds (init: 0.094, invoke: 0.119)

Expected behavior

az blob storage upload, instead of crashing, uploads the data that I provided.

Environment Summary

azure-cli 2.69.0

core 2.69.0
telemetry 1.1.0

Extensions:
ssh 2.0.6

Dependencies:
msal 1.31.2b1
azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3'
Config directory '/home/tobyc/.azure'
Extensions directory '/home/tobyc/.azure/cliextensions'

Python (Linux) 3.12.8 (main, Feb 5 2025, 06:40:01) [GCC 12.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @phire-tc, may I ask what is the format that the input data is in, I tried it in powershell with a .whl file and it works.

[phire-tc]

Interesting! I wonder if Python's sys.stdin.read() is different on Windows and Linux. I'm using Debian 12, and trying to upload an xz-compressed archive, which the file command reports to be "XZ compressed data, checksum CRC64".

Python's documentation for sys.stdin [1] indicates that it's a text file, and has a note that to read binary data from stdin you need to use the underlying buffer object: https://docs.python.org/3/library/sys.html#sys.stdin

[ArthurAttout]

Any updates on this ? Without it, I don't see any way to perform uploads without creating an intermediate file.