test(vm_extensions): add RunCommand v1/v2 boot validation test cases by tarunsingh-msft · Pull Request #4599 · microsoft/lisa

tarunsingh-msft · 2026-07-16T06:32:41Z

What

Adds minimal, self-contained boot validation test cases for the Run Command v1 and Run Command v2 VM extensions, mirroring the CustomScript boot validation test added in #4580.

RunCommandV1Tests.microsoft_cplat_core_runcommandlinux_boot_validation_test (lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py)
RunCommandV2Tests.microsoft_cplat_core_runcommandhandlerlinux_boot_validation_test (lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py)

Details

Installs the extension with a single inline command and no file/script uris — so it requires no storage account and no public blob access.
- v1 uses commandToExecute: "echo 'RCv1 boot validation success'".
- v2 uses source: { CommandId: "RunShellScript", script: "echo 'RCv2 boot validation success'" }.
Asserts provisioning_state == "Succeeded".
v1 removes the extension when done (RunCommandLinux is user-managed and deletable).
v2 does not delete the extension: RunCommandHandlerLinux is managed by the Compute Resource Provider and cannot be removed with a normal Delete VM Extension operation, so cleanup is handled by resource group teardown (same pattern as the existing v2 cases).
Publisher and type are read from runbook variables (extension_publisher, extension_type), defaulting to the respective Run Command extension. extension_version is required — the test is skipped if it is not set (same version-handling pattern as GenericVmExtension.verify_vm_extension_install_uninstall).
Both cases share a common helper, run_extension_boot_validation() in common.py, which reads publisher/type from runbook variables (with caller-provided defaults), requires extension_version (the test is skipped if it is not set), installs the extension, asserts provisioning succeeds, and optionally deletes it (cleanup=False for the CRP-managed v2). The helper is generic so future extensions can reuse it.
Deployed extension instance name follows <publisher>_<extension_type>_boot_validation_test

Testing

Ran end-to-end on Azure (Ubuntu 22.04, real VM deploy):

2026-07-17 12:20:24.886[28324][INFO] lisa.RootRunner RunCommandV1Tests.microsoft_cplat_core_runcommandlinux_boot_validation_test: PASSED
2026-07-17 12:20:24.886[28324][INFO] lisa.RootRunner     TOTAL    : 1
2026-07-17 12:20:24.886[28324][INFO] lisa.RootRunner     FAILED   : 0
2026-07-17 12:20:24.886[28324][INFO] lisa.RootRunner     PASSED   : 1
2026-07-17 12:20:24.886[28324][INFO] lisa.RootRunner     SKIPPED  : 0

2026-07-17 12:15:50.935[15568][INFO] lisa.RootRunner RunCommandV2Tests.microsoft_cplat_core_runcommandhandlerlinux_boot_validation_test: PASSED
2026-07-17 12:15:50.935[15568][INFO] lisa.RootRunner     TOTAL    : 1
2026-07-17 12:15:50.935[15568][INFO] lisa.RootRunner     FAILED   : 0
2026-07-17 12:15:50.935[15568][INFO] lisa.RootRunner     PASSED   : 1
2026-07-17 12:15:50.935[15568][INFO] lisa.RootRunner     SKIPPED  : 0

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Copilot

Pull request overview

Adds two new “boot validation” test cases under the VM extensions runtime test suites to validate successful provisioning of Run Command VM extensions using an inline command (no script/file URIs), avoiding storage account / public blob access requirements.

Changes:

Add a Run Command v1 boot-validation test that installs the extension with an inline commandToExecute, asserts provisioning_state == "Succeeded", and deletes the extension afterward.
Add a Run Command v2 boot-validation test that installs the extension with an inline RunShellScript, asserts provisioning_state == "Succeeded", and relies on RG teardown for cleanup (per RCv2 behavior).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py	Adds Run Command v1 boot validation test case using inline command and explicit extension cleanup.
lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py	Adds Run Command v2 boot validation test case using inline RunShellScript and verifying provisioning success.

tarunsingh-msft · 2026-07-16T08:54:08Z

@microsoft-github-policy-service agree company="Microsoft"

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

+                "case."
+            )
+
+        extension_name = f"{publisher}_{extension_type}_boot_validation_test"


+                "case."
+            )
+
+        extension_name = f"{publisher}_{extension_type}_boot_validation_test"


Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

+    publisher: str = variables.get("extension_publisher", default_publisher).strip()
+    extension_type: str = variables.get(
+        "extension_type", default_extension_type
+    ).strip()
+    version: str = variables.get("extension_version", "").strip()


+    try:
+        extension.normalize_type_handler_version(version)
+    except LisaException:
+        raise SkippedException(
+            f"Runbook variable 'extension_version'='{version}' is not a valid "
+            "'Major.Minor' or 'Major.Minor.Patch' version. Please set a valid "
+            "version in the runbook before running this test case."
+        )
+
+    if cleanup:
+        extension.delete(name=extension_name, ignore_not_found=True)
+
+    try:
+        log.info(f"Installing extension '{extension_name}'...")
+        result = extension.create_or_update(
+            name=extension_name,
+            publisher=publisher,
+            type_=extension_type,
+            type_handler_version=version,
+            auto_upgrade_minor_version=True,
+            settings=settings,
+            protected_settings=protected_settings or {},
+        )


Adds minimal, self-contained boot validation test cases for the Run Command v1 and v2 VM extensions, mirroring the CustomScript boot validation test (PR microsoft#4580). - RunCommandV1Tests.microsoft_azure_extensions_runcommandv1_boot_validation_test - RunCommandV2Tests.microsoft_azure_extensions_runcommandv2_boot_validation_test Each installs the extension with a single inline command and no file/script uris (no storage account or public blob access needed), asserts provisioning_state == Succeeded, then removes the extension. Publisher/type/version are read from runbook variables; the test is skipped when extension_version is unset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Collapse the extension_type assignment onto a single line to satisfy the Black formatter (BLK100), matching the flake8 CI check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

…lidation tests Reword the RCv1/RCv2 boot validation docstrings so the version handling matches the implementation: publisher and type default to the Run Command extension, while extension_version must be supplied via the runbook (the test is skipped otherwise). Addresses PR review feedback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

… publisher/type The certification functional-validation feature parses the test method name using the pattern <Publisher>_<Type>_boot_validation_test. RunCommand is published under Microsoft.CPlat.Core (types RunCommandLinux / RunCommandHandlerLinux), so the method names now reflect the actual publisher/type instead of the CustomScript publisher prefix: - microsoft_azure_extensions_runcommandv1_boot_validation_test -> microsoft_cplat_core_runcommandlinux_boot_validation_test - microsoft_azure_extensions_runcommandv2_boot_validation_test -> microsoft_cplat_core_runcommandhandlerlinux_boot_validation_test Publisher/type are unchanged in code (Microsoft.CPlat.Core, since 2023 PRs microsoft#2809/microsoft#2778) and match the deployed extension instance name. Re-validated on real Azure VMs: both cases PASSED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Consolidate the duplicated RunCommand v1/v2 boot validation flow into a single reusable helper run_extension_boot_validation() in common.py, as suggested in review. The helper reads publisher/type from runbook variables (with caller-provided defaults), requires extension_version (skips otherwise), installs the extension, asserts provisioning succeeds, and optionally deletes it (cleanup=False for CRP-managed RunCommand v2). A protected_settings parameter is included so future extensions can reuse it. Both cases re-validated on real Azure VMs after the refactor: PASSED (v1 eastus2, v2 eastus). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Add a guard in run_extension_boot_validation that validates extension_version is a 'Major.Minor' or 'Major.Minor.Patch' value (reusing AzureExtension.normalize_type_handler_version) and raises SkippedException instead of failing with a confusing Azure error when it is malformed. Addresses review feedback on version validation. Validated on Azure: valid versions (1.0/1.3) PASSED; malformed versions (1.0.beta / abc) SKIPPED with a clear message. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Address review nit: 'uris' -> 'URIs' in the RCv1/RCv2 boot validation test descriptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/common.py:95

Major: normalize_type_handler_version() is called only for validation, but the normalized Major.Minor install version isn’t used. If a runbook provides a patch version (e.g. 1.3.2), Azure extension install typically expects Major.Minor, so passing the raw version into create_or_update() can fail even though the validator accepts it. Capture the normalized version and pass that to type_handler_version.

    try:
        extension.normalize_type_handler_version(version)
    except LisaException:

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py:147

Major: This boot validation installs RunCommandHandlerLinux under a new VM extension name (<publisher>_<type>_boot_validation_test) while the suite already installs the same publisher+type under the fixed name RunCommandv2 (see _create_and_verify_extension_run around name="RunCommandv2", type_="RunCommandHandlerLinux"). Azure forbids multiple VM extensions with the same publisher+type on a VM; additionally, cleanup=False means this extra instance will remain and can break later RCv2 cases in the same run.

Consider reusing the suite’s canonical extension resource name (e.g. install/update RunCommandv2) for this test, or extend run_extension_boot_validation() to accept an explicit extension_name override and pass RunCommandv2 here.

        run_extension_boot_validation(
            node=node,
            log=log,
            variables=variables,
            default_publisher="Microsoft.CPlat.Core",
            default_extension_type="RunCommandHandlerLinux",
            settings={
                "source": {
                    "CommandId": "RunShellScript",
                    "script": "echo 'RCv2 boot validation success'",
                }
            },
            # RunCommand v2 is CRP-managed and cannot be deleted; resource
            # group teardown handles cleanup.
            cleanup=False,
        )

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py:129

Major: This boot validation installs RunCommandLinux under a new VM extension name (<publisher>_<type>_boot_validation_test), but the rest of the suite uses the fixed name RunCommandv1 for the same publisher+type (see _create_and_verify_extension_run: extension_name = "RunCommandv1", type_="RunCommandLinux"). Because Azure forbids multiple VM extensions with the same publisher+type on a VM, this test can fail or interfere depending on execution order.

Consider reusing RunCommandv1 as the extension instance name for this test (or extend run_extension_boot_validation() to accept an explicit extension_name override and pass RunCommandv1 here), and/or adopt the same “delete existing same handler type” cleanup pattern as GenericVmExtension.verify_vm_extension_install_uninstall.

        run_extension_boot_validation(
            node=node,
            log=log,
            variables=variables,
            default_publisher="Microsoft.CPlat.Core",
            default_extension_type="RunCommandLinux",
            settings={"commandToExecute": "echo 'RCv1 boot validation success'"},
        )

…review Address review feedback on the RunCommand boot validation tests: - Coerce runbook variables to str before strip(), so a non-string YAML value (e.g. extension_version: 1.3 parsed as a float) does not crash the helper. - Use the normalized Major.Minor value from normalize_type_handler_version() for type_handler_version, so a Major.Minor.Patch runbook value installs correctly instead of being passed through raw. - Set maturity="preview" on both boot validation cases so they are excluded from default stable-only runs until fully validated, consistent with the CustomScript boot validation test (microsoft#4606) and the test maturity model (microsoft#4586). Re-validated on real Azure VMs after these changes: both cases PASSED (v1 centralus, v2 eastus2). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py:147

This boot-validation case installs RunCommandHandlerLinux via run_extension_boot_validation() with cleanup=False, which leaves the extension deployed under a different name than the rest of the suite. The suite’s other test cases install the same publisher/type under name "RunCommandv2" (see _create_and_verify_extension_run at run_commandv2.py:53-61). Because Azure disallows multiple extensions with the same handler type, running this case together with the rest of the suite can produce handler-type conflicts. Prefer reusing the existing suite extension name for v2 (and making the helper idempotent against an already-installed handler) rather than creating a second instance name.

        run_extension_boot_validation(
            node=node,
            log=log,
            variables=variables,
            default_publisher="Microsoft.CPlat.Core",
            default_extension_type="RunCommandHandlerLinux",
            settings={
                "source": {
                    "CommandId": "RunShellScript",
                    "script": "echo 'RCv2 boot validation success'",
                }
            },
            # RunCommand v2 is CRP-managed and cannot be deleted; resource
            # group teardown handles cleanup.
            cleanup=False,
        )

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py:129

This boot-validation case installs RunCommandLinux under a new name via run_extension_boot_validation(). The rest of the RunCommand v1 suite installs the same handler type under the fixed name "RunCommandv1" (run_commandv1.py:41-54) and does not remove it after each case. If this boot-validation test runs after another v1 case in the same VM, Azure may reject installing the same publisher/type under a second name. Reusing the existing "RunCommandv1" instance name (or deleting any pre-existing extensions matching the same publisher+type before installing) would avoid ordering-dependent failures.

        run_extension_boot_validation(
            node=node,
            log=log,
            variables=variables,
            default_publisher="Microsoft.CPlat.Core",
            default_extension_type="RunCommandLinux",
            settings={"commandToExecute": "echo 'RCv1 boot validation success'"},
        )

Address review feedback (LiliDeng): the boot validation callers never pass protected_settings, so remove the unused parameter from run_extension_boot_validation(). It can be reintroduced if a future extension actually needs protected settings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py:128

Per LISA metadata conventions, each test case should include an explicit requirement= in @TestCaseMetadata (even if it matches the suite requirement) so the case is self-describing and tooling can reason about per-case constraints.

        priority=5,
        maturity="preview",
    )

lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py:118

Per LISA metadata conventions, each test case should include an explicit requirement= in @TestCaseMetadata (even if it matches the suite requirement) so the case is self-describing and tooling can reason about per-case constraints.

        priority=5,
        maturity="preview",
    )

+    publisher: str = str(
+        variables.get("extension_publisher", default_publisher)
+    ).strip()


+    extension_type: str = str(
+        variables.get("extension_type", default_extension_type)
+    ).strip()


Copilot AI review requested due to automatic review settings July 16, 2026 06:32

tarunsingh-msft requested review from LiliDeng, johnsongeorge-w and mayankdaruka-msft as code owners July 16, 2026 06:32

Copilot started reviewing on behalf of tarunsingh-msft July 16, 2026 06:33 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py Outdated

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py Outdated

tarunsingh-msft closed this Jul 16, 2026

tarunsingh-msft reopened this Jul 16, 2026

Copilot AI review requested due to automatic review settings July 16, 2026 09:03

Copilot started reviewing on behalf of tarunsingh-msft July 16, 2026 09:04 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py Outdated

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py Outdated

Copilot AI review requested due to automatic review settings July 16, 2026 09:27

Copilot started reviewing on behalf of tarunsingh-msft July 16, 2026 09:28 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

lubaihua33 reviewed Jul 17, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py Outdated

lubaihua33 reviewed Jul 17, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py Outdated

Copilot AI review requested due to automatic review settings July 17, 2026 09:12

Copilot started reviewing on behalf of tarunsingh-msft July 17, 2026 09:13 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py Outdated

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py Outdated

Copilot AI review requested due to automatic review settings July 17, 2026 12:23

Copilot started reviewing on behalf of tarunsingh-msft July 17, 2026 12:23 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/common.py Outdated

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv1.py Outdated

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/run_commandv2.py

lubaihua33 previously approved these changes Jul 17, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 17, 2026 14:57

tarunsingh-msft dismissed lubaihua33’s stale review via b2887d9 July 17, 2026 14:57

Copilot started reviewing on behalf of tarunsingh-msft July 17, 2026 14:58 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

tarunsingh-msft requested a review from lubaihua33 July 17, 2026 15:06

tarunsingh-msft added 6 commits July 20, 2026 00:54

docs(vm_extensions): capitalize URIs in boot validation docstrings

6870480

Address review nit: 'uris' -> 'URIs' in the RCv1/RCv2 boot validation test descriptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c8bcd2c-8f65-4576-a13b-870fb7163f0f

Copilot AI review requested due to automatic review settings July 20, 2026 07:58

tarunsingh-msft force-pushed the singhtarun/runcommand-boot-validation-tests branch from b2887d9 to 86e1caa Compare July 20, 2026 07:58

Copilot started reviewing on behalf of tarunsingh-msft July 20, 2026 07:59 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 20, 2026 08:32

tarunsingh-msft force-pushed the singhtarun/runcommand-boot-validation-tests branch from 86e1caa to 7c1e060 Compare July 20, 2026 08:32

Copilot started reviewing on behalf of tarunsingh-msft July 20, 2026 08:33 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/common.py

LiliDeng reviewed Jul 20, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/vm_extensions/runtime_extensions/common.py Outdated

Copilot AI review requested due to automatic review settings July 20, 2026 08:46

Copilot started reviewing on behalf of tarunsingh-msft July 20, 2026 08:47 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

LiliDeng approved these changes Jul 20, 2026

View reviewed changes

LiliDeng merged commit 689fb7b into microsoft:main Jul 20, 2026
27 checks passed

Uh oh!

Conversation

tarunsingh-msft commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Details

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

tarunsingh-msft commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tarunsingh-msft commented Jul 16, 2026 •

edited

Loading