serial console broken after live migration

[Klaas-]

Hi,
it seems that the serial console breaks after a live migration when using trusted launch (with secure boot and vtpm). Support told me there are no plans to fix this (case 2412030050001444) , is that correct?

Greetings
Klaas

[Klaas-]

For anyone searching for this issue:
serial console being unavailable leads to several issues, at the heart of this issue is that /dev/console is no longer accepting writes and just hanging forever.

we noticed it because azure extensions are starting to hang like the backup extension because they are writing to /dev/console in a matter that is blocking to backups, so that means snapshots are now taking multiple hours instead of 10 minutes (I am guessing they do not have sensible timeouts). But it also leads to a reboot hanging because linux seemingly wants to finish the writes before rebooting (or tries to write to it itself, I have not debugged that reboot issue).

It also leads to a loss of the serial console log prior to the live migration.

I am guessing this is going to hit more people once they start adopting trusted launch....

[mbifeld]

Hi @Klaas-, thank you for reporting this issue. This is the first instance of this issue occurring that our product team has heard about. We have documented this as a known issue thanks to your reporting. I'm glad that rebooting does allow serial console to be back online, but it is unfortunate that reboots hang due to linux wanting to finish the writes. It's also unfortunate that the serial logs pre-migration are lost.

It is not that we do not plan to address this issue, but rather that we cannot provide an eta at this time. It appears to be a low-level issue since it affects serial logs too (which is a separate service) - so it is likely an issue that will take partnerships across several services to address. We will reach out if we require more information from your end when debugging. Thank you.

[Klaas-]

Hi @mbifeld ,
so good to hear that you plan to address this software defect. If this is not a short-term (less than 6 months) fix, then the question is how do you want to deal with this in the meantime. I would personally like to see Azure stop live migrations on the VMs that use trusted launch+boot diagnostics/serial console, the operation of live migration is obviously not working and breaks the machines, Azure should not break machines actively. This does have the obvious downside of live migrations not happening anymore, but it's one I can live with, to fix the issue I have to redeploy/power off the VM anyways.

[Klaas-]

Also I think you should check and extend your QA, this issue seems to happen on every live migration with boot diagnostics+trusted launch. So I am guessing this is something that has never been tested in QA.

[mbifeld]

Good feedback @Klaas-. The decision on Azure allowing live migrations isn't for me to make, but I'll ensure the appropriate team is aware of this issue - including your point about ensuring live migration for VMs with boot diagnostics + Trusted Launch is in the testing pipeline.

[Klaas-]

Also someone should get this problem into monitoring. Resource Health does not understand this problem it keeps showing the affected VMs post live migration as "Green" and "Available" whereas they should most likely be degraded or something like that.

[Klaas-]

Crosslinking Azure/azure-linux-extensions#2039 now extensions start working around the problem instead of MSFT fixing it at the source -- I mean great for backup, it should never hang because of logging not working :)

[Klaas-]

So this issue is like 50% fixed right now, after a live migration serial console and boot diagnostics are working. However a live migration loses all boot diagnostics logs. This should only happen on the 1GB rollover.

[Klaas-]

After a full year a final fix has been developed by microsoft and is in rollout. I do not have details about the rollout process.. :) I'll close this in the hopes that we're talking about weeks and not months/years of rollout :)