Add sled could not complete

[askfongjojo]

This is on rack2, after a mupdate (to get past a failed reconfigurator-driven update) and blueprint archival. Two new sleds in cubby 2 and 3 were updated to the same version:

root@oxz_switch0:~# pilot host ls
CUBBY IP                        SERIAL      IMAGE
2     fe80::aa40:25ff:fe04:c96  BRM22250001 ci 5eb1337/4e5b80e 2026-01-21 22:22
3     fe80::aa40:25ff:fe04:412  BRM13250012 ci 5eb1337/4e5b80e 2026-01-21 22:22
7     fe80::aa40:25ff:fe04:6d6  BRM27230045 ci 5eb1337/4e5b80e 2026-01-21 22:22
8     fe80::aa40:25ff:fe04:3d5  BRM44220011 ci 5eb1337/4e5b80e 2026-01-21 22:22
9     fe80::aa40:25ff:fe04:357  BRM44220005 ci 5eb1337/4e5b80e 2026-01-21 22:22
10    fe80::aa40:25ff:fe04:3d4  BRM42220009 ci 5eb1337/4e5b80e 2026-01-21 22:22
11    fe80::aa40:25ff:fe04:191  BRM42220006 ci 5eb1337/4e5b80e 2026-01-21 22:22
12    fe80::aa40:25ff:fe04:393  BRM42220057 ci 5eb1337/4e5b80e 2026-01-21 22:22
13    fe80::aa40:25ff:fe04:1d1  BRM42220018 ci 063828b/10bf4ba 2025-10-09 02:10
14    fe80::aa40:25ff:fe04:195  BRM42220051 ci 5eb1337/4e5b80e 2026-01-21 22:22
16    fe80::aa40:25ff:fe04:352  BRM42220014 ci 5eb1337/4e5b80e 2026-01-21 22:22
17    fe80::aa40:25ff:fe04:192  BRM42220017 ci 5eb1337/4e5b80e 2026-01-21 22:22
21    fe80::aa40:25ff:fe04:353  BRM42220031 ci 5eb1337/4e5b80e 2026-01-21 22:22
23    fe80::aa40:25ff:fe04:395  BRM42220016 ci 5eb1337/4e5b80e 2026-01-21 22:22
25    fe80::aa40:25ff:fe04:354  BRM44220010 ci 5eb1337/4e5b80e 2026-01-21 22:22

Component versions before adding sled 2:

root@oxz_switch0:~# omdb nexus update-status
Count of each component type by system version:

                  |18.0.0-0.ci+git5eb13372380 
------------------+---------------------------
RoT bootloader    |15                         
RoT               |15                         
SP                |15                         
Host OS (phase 1) |12                         
Host OS (phase 2) |12                         
Zone              |149   

root@oxz_switch0:~# omdb nexus sleds list-uninitialized
RACK_ID                              CUBBY SERIAL      PART        REVISION 
de608e01-b8e4-4d93-b972-a7dbed36dd22 2     BRM22250001 913-0000023 1        
de608e01-b8e4-4d93-b972-a7dbed36dd22 3     BRM13250012 913-0000023 1        
de608e01-b8e4-4d93-b972-a7dbed36dd22 13    BRM42220018 913-0000019 6 

An attempt to add sled 2 ended with an error

root@oxz_switch0:~# omdb nexus sleds add BRM22250001 913-0000023 -w
Error: adding sled

Caused by:
    0: Communication Error: error sending request for url (http://[fd00:1122:3344:104::56]:12232/sleds/add)
    1: error sending request for url (http://[fd00:1122:3344:104::56]:12232/sleds/add)
    2: operation timed out

The sled still got added to the cluster (the sled is listed in omdb db sleds and gone from list-uninitialized), and also has the component versions set:

root@oxz_switch0:~# omdb nexus update-status
Count of each component type by system version:

                  |18.0.0-0.ci+git5eb13372380 
------------------+---------------------------
RoT bootloader    |16                         
RoT               |16                         
SP                |16                         
Host OS (phase 1) |13                         
Host OS (phase 2) |13                         
Zone              |149

The current blueprint executor status is as follows:

root@oxz_switch0:~# omdb nexus blueprints list 2>/dev/null | tail -5
      2d05edea-be41-47e3-a99f-4604c6d61a9f 06767a6b-f1c3-45ca-bb4f-0317487e7c16 2026-01-22T20:05:23.387Z 
      162afbd6-5a20-446f-abf5-fcc83a5b6b96 2d05edea-be41-47e3-a99f-4604c6d61a9f 2026-01-23T05:58:53.302Z 
      d5d2b2cd-12e8-4854-9faf-391a0db1ae9e 162afbd6-5a20-446f-abf5-fcc83a5b6b96 2026-01-23T05:59:36.847Z 
      0b924bd4-d860-4767-a99d-0703e669a6f0 d5d2b2cd-12e8-4854-9faf-391a0db1ae9e 2026-01-23T05:59:38.121Z 
* yes 94192783-0bf5-4c23-90c1-ba223c040d49 0b924bd4-d860-4767-a99d-0703e669a6f0 2026-01-23T05:59:42.778Z 

root@oxz_switch0:~# omdb nexus background-tasks show blueprint_planner
task: "blueprint_planner"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 1774, triggered by a dependent task completing
    started at 2026-01-23T06:14:27.977Z (23s ago) and ran for 777ms
    plan unchanged from parent 94192783-0bf5-4c23-90c1-ba223c040d49
    note: 249/5000 blueprints in database
planning report:
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - current target release generation (35) is lower than minimum required by blueprint (36)
* zone updates waiting on zone add blockers
* will ensure cockroachdb setting: "22.1"

The sled only has a global zone at this point and I've marked it non-provisionable for now.

[askfongjojo]

Reconfigurator history:

root@oxz_switch0:~# omdb reconfigurator history 2>/dev/null | tail -10
36490 2026-01-22T01:10:16.980Z f4fc3831-6998-45e2-a578-bc53f6bc1773  enabled: update Sled 16 (BRM42220014) SP to 1.0.57
36491 2026-01-22T01:14:21.060Z 3ddfc3f2-1dc6-486d-8acb-c0eeb2a1b188  enabled: blueprint details no longer available
36492 2026-01-22T01:16:42.067Z 5bdda519-30be-4f31-b60a-2c92c81e3c0a  enabled: blueprint details no longer available
36493 2026-01-22T01:24:29.947Z 8145dc1f-4338-472e-b8ae-e401ff10f5f6  enabled: blueprint details no longer available
36494 2026-01-22T20:04:47.307Z 06767a6b-f1c3-45ca-bb4f-0317487e7c16  enabled: blueprint details no longer available
36495 2026-01-22T20:05:23.842Z 2d05edea-be41-47e3-a99f-4604c6d61a9f  enabled: sled 0c7011f7-a4bf-4daf-90cc-1c2410103300: performed 12 noop zone image source updates, sled 0c7011f7-a4bf-4daf-90cc-1c2410103300: noop updated host phase 2 to Artifact: both slot A and slot B, sled 2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa: performed 11 noop zone image source updates, sled 2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa: noop updated host phase 2 to Artifact: both slot A and slot B, sled 5f6720b8-8a31-45f8-8c94-8e699218f28b: performed 12 noop zone image source updates, sled 5f6720b8-8a31-45f8-8c94-8e699218f28b: noop updated host phase 2 to Artifact: both slot A and slot B, sled 71def415-55ad-46b4-ba88-3ca55d7fb287: performed 13 noop zone image source updates, sled 71def415-55ad-46b4-ba88-3ca55d7fb287: noop updated host phase 2 to Artifact: both slot A and slot B, sled 7b473a3b-4ec2-4b58-8376-9b3cb68d1392: performed 11 noop zone image source updates, sled 7b473a3b-4ec2-4b58-8376-9b3cb68d1392: noop updated host phase 2 to Artifact: both slot A and slot B, sled 7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c: performed 13 noop zone image source updates, sled 7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c: noop updated host phase 2 to Artifact: both slot A and slot B, sled 87c2c4fc-b0c7-4fef-a305-78f0ed265bbc: performed 12 noop zone image source updates, sled 87c2c4fc-b0c7-4fef-a305-78f0ed265bbc: noop updated host phase 2 to Artifact: both slot A and slot B, sled a2adea92-b56e-44fc-8a0d-7d63b5fd3b93: performed 12 noop zone image source updates, sled a2adea92-b56e-44fc-8a0d-7d63b5fd3b93: noop updated host phase 2 to Artifact: both slot A and slot B, sled b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88: performed 13 noop zone image source updates, sled b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88: noop updated host phase 2 to Artifact: both slot A and slot B, sled db183874-65b5-4263-a1c1-ddb2737ae0e9: performed 13 noop zone image source updates, sled db183874-65b5-4263-a1c1-ddb2737ae0e9: noop updated host phase 2 to Artifact: both slot A and slot B, sled dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd: performed 13 noop zone image source updates, sled dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd: noop updated host phase 2 to Artifact: both slot A and slot B, sled f15774c1-b8e5-434f-a493-ec43f96cba06: performed 14 noop zone image source updates, sled f15774c1-b8e5-434f-a493-ec43f96cba06: noop updated host phase 2 to Artifact: both slot A and slot B
36496 2026-01-23T05:58:53.647Z 162afbd6-5a20-446f-abf5-fcc83a5b6b96  enabled: 
36497 2026-01-23T05:59:37.287Z d5d2b2cd-12e8-4854-9faf-391a0db1ae9e  enabled: sled 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75: noop updated host phase 2 to Artifact: both slot A and slot B
36498 2026-01-23T05:59:38.452Z 0b924bd4-d860-4767-a99d-0703e669a6f0  enabled: 
36499 2026-01-23T05:59:43.147Z 94192783-0bf5-4c23-90c1-ba223c040d49  enabled: sled 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75: noop updated host phase 2 to Artifact: both slot A and slot B

The archived blueprints can be found in /staff/dock/rack2/upgrade-20260121.

[askfongjojo]

Turned out that it was because I didn't set the target release again after mupdate, even though it is the same version.
Sled 2 got its ntp and crucible zones after I ran

oxide system update target-release update --system-version "18.0.0-0.ci+git5eb13372380"

This isn't something user can do via the console because the target set option is disabled when the release is already the target. The above can only be invoked via the CLI.

Another oddity is that a similar planner message showed up when I added another new sled (sled 3):

root@oxz_switch0:~# omdb nexus background-tasks show blueprint_planner
task: "blueprint_planner"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 8096, triggered by a dependent task completing
    started at 2026-01-24T18:39:31.539Z (17s ago) and ran for 649ms
    plan unchanged from parent ea0d5907-ec5d-45c2-b889-e0d31463094e
    note: 255/5000 blueprints in database
planning report:
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - current target release generation (36) is lower than minimum required by blueprint (37)
* zone updates waiting on zone add blockers
* will ensure cockroachdb setting: "22.1"

I set the target release again to "18.0.0-0.ci+git5eb13372380". After that sled 3 also got its omicron zones.

@sunshowers - Is setting the target release step required whenever a new sled is added? The one I forgot to do after mupdate makes sense because nexus had an override from wicket; but adding a new sled is fully controlled by nexus so I'd expect it to take care of updating the generation?

[askfongjojo]

Btw, I also received the same communication error when adding sled 3, likely a client side timeout from omdb only (which still allowed the sled addition to be initiated). I'll document that as a known issue in the add-sled runbook for now. Perhaps omdb can use a longer timeout for this particular operation (or provide a --timeout override). I'll check how the issue may show up when using the operator API to add a sled.

[sunshowers]

Not at my computer right now or until tomorrow afternoon, but was sled 3 in a pristine state or was it reading an old mupdate override on accident (something that I believe full TQ solves?)

The current target release generation is operator-controlled policy so it would be incorrect for Nexus to bump it. So the question is why the minimum target release generation (which is Nexus-controlled state) got bumped — that should only happen if a new mupdate override is detected. (There could be a bug here.)

[askfongjojo]

I included the two new sleds in the cubby list within the same wicket CLI command so they should have identical mupdate artifacts as the existing sleds. The two sleds weren't updated again afterwards. To summarize, the sequence of events is:

1. Uploaded trust root and the TUF repo for 18.0.0-0.ci+git5eb13372380, set target release. Sled 2 and 3 were not in the cluster yet; a subset of sleds got through SP and host OS update but couldn't proceed when sled 16 found the dendrite API version mismatch.
2. Used the same TUF repo to mupdate sled 16 to 18.0.0-0.ci+git5eb13372380 (via wicket TUI), then mupdated sled 2, 3, and all other sleds (in one wicket CLI call).
3. Re-uploaded the TUF repo and got an "already exists" error. (did not set target release afterwards)
4. Executed omdb nexus sleds add <sled 2 serial> (sled showed up in omdb db sleds but had no omicron zones because of generation mismatch).
5. Ran oxide system update target-release update --system-version "18.0.0-0.ci+git5eb13372380". (omicron zones provisioned on sled 2)
6. Repeated step 4 for sled 3. (same outcome - sled added but no omicron zones)
7. Repeated step 5. (same outcome - omicron zones provisioned)

[askfongjojo]

We concluded that the behavior is expected and probably has been like that since at least v17 (i.e., not a regression). We just never noticed this issue as we haven't tried adding two sleds in succession between software updates. The add-sled runbook has been updated to include the set target release step, as well as the communication error (client-side timeout) issue.

[andrewjstone]

Adding sleds will proceed via real trust quorum now.

[andrewjstone]

I closed it when I possibly shouldn't have. Leaving for @askfongjojo.