Skip to content

Commit a1d29ce

Browse files
SebasRG23SebasRG23
committed
Update Incompatible extensions TSG for 2602
1 parent f522562 commit a1d29ce

1 file changed

Lines changed: 89 additions & 23 deletions

File tree

‎TSG/Update/UpdatePrerequisites-Failure-Remove-Incompatible-Extensions.md‎

Lines changed: 89 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,101 @@
1-
# [2503] UpdatePreRequisites Failure: Update fails with "invalid escape sequence" error because of extension incompatibility
1+
# Update Failure: Incompatible AZ CLI extensions (UpdatePreRequisites / AzCLIKnownIncompatibleExtensions)
22

3-
# Symptoms
4-
An update to 10.2503.0.13 fails with errors like below:
3+
## Symptoms
54

6-
*C:\CloudContent\AzCliExtensions\hybridaks\azext_hybridaks\vendored_sdks\hybridaks\models\_models_py3.py:3800: SyntaxWarning: invalid escape sequence '\W'*
5+
An update fails during **UpdatePreRequisites** or the **pre-update environment validation** phase with one of the following error patterns:
76

8-
*C:\CloudContent\AzCliExtensions\connectedmachine\azext_connectedmachine\aaz\latest\connectedmachine\_delete.py:54: SyntaxWarning: invalid escape sequence '\.'*
7+
- **Python SyntaxWarning during UpdatePreRequisites:**
8+
```
9+
C:\CloudContent\AzCliExtensions\hybridaks\azext_hybridaks\vendored_sdks\hybridaks\models_models_py3.py:3800: SyntaxWarning: invalid escape sequence '\W'
10+
C:\CloudContent\AzCliExtensions\connectedmachine\azext_connectedmachine\aaz\latest\connectedmachine_delete.py:54: SyntaxWarning: invalid escape sequence '\.'
11+
C:\CloudContent\AzCliExtensions\azurestackhci\azext_azurestackhci\generated_validators.py:18: SyntaxWarning: invalid escape sequence '\['
12+
```
913

10-
*C:\CloudContent\AzCliExtensions\azurestackhci\azext_azurestackhci\generated\_validators.py:18: SyntaxWarning: invalid escape sequence '\['*
14+
- **Pre-update health check failure**: the environment validator reports a failed test similar to:
15+
```
16+
Name: AzStackHci_ARBStack_AzCLIKnownIncompatibleExtensions
17+
Status: FAILURE
18+
Description: Detect known incompatible Azure CLI extensions that should be removed.
19+
Remediation: Please remove unsupported Azure CLI extensions to ensure compatibility (<extension-name>).
20+
Detail: Known incompatible Azure CLI extensions detected: <extension-name>.
21+
These extensions will cause issues during the update process.
22+
```
23+
> **Note:** The `TargetResourceName` field in the health check result identifies the specific node(s) where the incompatible extension was detected.
1124
12-
# Issue Validation
13-
This TSG is only for updates to 10.2503.0.13.
25+
Both patterns indicate that AZ CLI extensions **not managed by Azure Local** are present on one or more nodes and may be incompatible with the AZ CLI core version shipped with the target release.
1426

15-
Below is the list of AzCli extensions which are expected to be installed for 10.2503.0.13 updates:
16-
| Az CLI Extension Name | RequiredVersion |
17-
| ------------- | ------------- |
18-
| arcappliance | 1.3.0 |
19-
| k8s-extension | 1.4.5 |
20-
| customlocation | 0.1.3 |
21-
| stack-hci-vm | 1.4.2 |
27+
## Root Cause
2228

23-
If there are any extensions installed that are not in the above list, it could be causing the failures described in this TSG.
29+
AZ CLI uses the `AZURE_EXTENSION_DIR` environment variable to locate its extensions directory. Every `az` command automatically loads **all** extensions present in that directory, regardless of whether they are managed by Azure Local or not.
2430

25-
Run the following command on all nodes to verify the Azure CLI extensions installed on each node:
31+
Each Azure Local release ships a ValidatedRecipe that defines the exact set of AZ CLI extensions (and their versions) required for the solution. These **managed extensions** are tested for compatibility with the AZ CLI core version included in each release. During UpdatePreRequisites, MocArb installs or upgrades them.
2632

27-
`az version`
33+
The problem occurs when **unmanaged extensions** (extensions that are not part of the ValidatedRecipe) are also present in the extensions directory (e.g., manually installed by a user, or left over from other tooling). AZ CLI core frequently introduces breaking changes between versions, particularly in the bundled Python runtime. Because Azure Local does not manage these extensions, they may be incompatible with the newer CLI core shipped in the update. When the CLI loads them:
2834

29-
Example for az version output (we are only focusing on the "extensions" section):
30-
![image.png](./images/azversionexampleoutput.png)
35+
- **SyntaxWarnings / SyntaxErrors** occur because the newer Python runtime rejects deprecated syntax in older extension code.
36+
- **Action plan failures** occur because CLI commands invoked by MocArb during the update return unexpected errors or crash due to the incompatible extension being loaded.
3137

32-
# Mitigation Details
38+
The pre-update environment validator (`AzStackHci_ARBStack_AzCLIKnownIncompatibleExtensions`) proactively detects known incompatible extensions and blocks the update until they are removed.
3339

34-
If there are any extensions installed that are not in the list, run below command on all nodes to remove the unexpected extension:
35-
`az extension remove --name "<extension name>"`
40+
## Applies To
41+
42+
All Azure Local releases. This is not specific to any single build version.
43+
44+
## Issue Validation
45+
46+
### Step 1: Get expected extensions from the ValidatedRecipe
47+
48+
On any node, run:
49+
```powershell
50+
# Load the ValidatedRecipe for the target release
51+
$vcNugetPath = Get-ASArtifactPath -NugetName 'Microsoft.AzureStack.VersionControl.Operations'
52+
Import-Module "$vcNugetPath\content\Scripts\VersionControlHelpers.psm1" -Force -DisableNameChecking
53+
$recipePath = Get-ValidatedRecipeJsonFilePath -TagsToInclude @('Azure')
54+
$recipe = Get-Content $recipePath -Raw | ConvertFrom-Json
55+
56+
# Filter to AZ CLI core and CLI extensions only
57+
$cliRecipe = $recipe | Where-Object { $_.Type -in @('AzCli', 'AzCliExtension') }
58+
59+
# Extract short extension names (last segment of nuget name) to match az version output
60+
$cliRecipe | Select-Object @{N='ExtensionName';E={ $_.Name.Split('.')[-1] }}, RequiredVersion, Type | Format-Table -AutoSize
61+
```
62+
63+
Example output:
64+
```
65+
ExtensionName RequiredVersion Type
66+
------------- --------------- ----
67+
AzureCLI 2.81.0 AzCli
68+
arcappliance 1.7.0 AzCliExtension
69+
k8s-extension 1.7.0 AzCliExtension
70+
customlocation 0.1.4 AzCliExtension
71+
stack-hci-vm 1.12.0 AzCliExtension
72+
```
73+
74+
### Step 2: Compare with installed extensions on each node
75+
76+
Run the following on **every node** in the cluster (the unmanaged extension may only be present on some nodes):
77+
78+
```powershell
79+
az version --output json --only-show-errors | ConvertFrom-Json | Select-Object -ExpandProperty extensions
80+
```
81+
82+
Example output:
83+
```
84+
arcappliance : 1.7.0
85+
connectedmachine : 1.1.0
86+
customlocation : 0.1.4
87+
k8s-extension : 1.7.0
88+
stack-hci-vm : 1.12.0
89+
```
90+
91+
Compare this against the expected list from Step 1. Any extension present here that is **not** in the ValidatedRecipe (e.g., `connectedmachine`, `azurestackhci` in the example above) is an **unmanaged extension** and should be removed before updating. These can be added back once the Solution Update completes.
92+
93+
## Mitigation
94+
95+
Remove any **unmanaged extension** (not listed in the ValidatedRecipe) from **every node** in the cluster:
96+
97+
```powershell
98+
az extension remove --name "<extension-name>" --only-show-errors
99+
```
100+
101+
Repeat for each unmanaged extension identified in Step 2, on every affected node. Then resume or retry the update. The UpdatePreRequisites step will handle installing the correct managed extensions automatically.

0 commit comments

Comments
 (0)