az vm run-command create doesn't return even the script is finished

[tlaukkanen]

Describe the bug

I'm running PowerShell scripts on Windows VMs with az vm run-command create ... command as part of our DevOps pipelines. I'm running those on multiple similar VMs but the command doesn't return randomly on many VMs even though the script has finished on VM. I'm logging the execution and PowerShell script does call exit 0 as last step but still the run-command hangs for some reason and eventually times out in DevOps. I can see with az vm run-command show ... that command is still in
"executionState": "Running" and "provisioningState": "Creating" states.

Related command

az vm run-command create \
  --name MyScript \
  --vm-name vm-server1 \
  --resource-group rg-myservers \
  --run-as-user mylocaluser \
  --run-as-password <pwd> \
  --script "PowerShell -ExecutionPolicy Unrestricted -NonInteractive -File C:\temp\myscript.ps1 -Foo:Bar"

Errors

I don't get errors but the command hangs for over an hour.
I can see the status with separate command:

az vm run-command show --name MyScript --resource-group rg-myservers --vm-name vm-server1 --expand instanceView

{
  "asyncExecution": false,
  "errorBlobUri": null,
  "id": "/subscriptions/.../resourceGroups/rg-myservers/providers/Microsoft.Compute/virtualMachines/vm-server1/runCommands/MyScript",
  "instanceView": {
    "endTime": null,
    "error": null,
    "executionMessage": null,
    "executionState": "Running",
    "exitCode": 0,
    "output": "...Script finished. Exiting the script.",
    "startTime": "2024-01-11T06:56:23.575284+00:00",
    "statuses": null
  },
  "location": "westeurope",
  "name": "MyScript",
  "outputBlobUri": null,
  "parameters": null,
  "protectedParameters": null,
  "provisioningState": "Creating",
  "resourceGroup": "rg-myservers",
  "runAsPassword": null,
  "runAsUser": "mylocaluser",
  "source": {
    "commandId": null,
    "script": "PowerShell -ExecutionPolicy Unrestricted -NonInteractive -File C:\\temp\\myscript.ps1 -Foo:Bar",
    "scriptUri": null
  },
  "tags": null,
  "type": "Microsoft.Compute/virtualMachines/runCommands"
}

Issue script & Debug output

Expected behavior

I would expect the az vm run-command create ... to return quite quickly after the given script is completed.

Environment Summary

azure-cli                         2.55.0 *

core                              2.55.0 *
telemetry                          1.1.0

Extensions:
account                            0.2.5
authV2                             0.1.1
azure-devops                      0.25.0

Additional context

I'm not able to delete the command either so it will block our DevOps pipelines for over an hour before command can be run again. I have also tried to delete the command but that also might take an hour to complete. For example:

> az vm run-command delete --name MyScript --resourcegroup rg-myservers --vm-name vm-server1 --yes --verbose
Command ran in 2950.874 seconds (init: 0.437, invoke: 2950.437)

[azure-client-tools-bot-prd]

Hi @tlaukkanen,

2.55.0 is not the latest Azure CLI(2.56.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[mpagot]

I'm able to reproduce the same in a different context.

azure-cli                         2.61.0 *

core                              2.61.0 *
telemetry                          1.1.0

Extensions:
account                            0.2.5
interactive                        0.5.3
storage-preview                  1.0.0b1

Dependencies:
msal                              1.28.0
azure-mgmt-resource               23.1.1

Python location '/home/<USER>/bin/.venv_az/bin/python3.12'
Extensions directory '/home/<USER>/.azure/cliextensions'

Python (Linux) 3.12.3 (main, Apr 10 2024, 14:51:56) [GCC]

Command I'm running is

vm run-command create \
  --run-command-name "awaitCloudInitIsDone" \
  -g $MY_GROUP \
  --vm-name $THIS_VM \
  --async-execution "false" \
  --run-as-user $MY_USERNAME \
  --timeout-in-seconds $AZ_CLOUTINIT_TIMEOUT \
  --script "sudo cloud-init status --wait"

I can connect to the same VM with ssh, short after the VM creation, and run the same. I can see the command executed with ssh to end, so the one executed with run-command should be over too but the API never return, even not after the --timeout-in-seconds.
In my experiments AZ_CLOUTINIT_TIMEOUT is set to 3600secs.