Issue with UFS storages

[fhost]

Hello,

I have a probem with version 0.14.3 (ubuntu resolute version), that I don't have with 0.11.1 (ubuntu noble version).

On my device, I have a UFS storage, split into multiple LUNs. This makes the storage appear as multiple disks (sda, sdb, etc...).
Of course, all those disks have the same wwid (as it is the same storage). So multipath tries to handle them.

This brings 2 issues.

1. It will find sda, wwid X, then sdb, wwid X, so it will make a map. But sda won't be added, as the size differ. But the map is still created anyway. This has two bad side effect:

   • Puts wwid as a valid wwid in /etc/multipath/wwids, even if it is not;
   • Makes multipath -u /dev/sda # or sdb return 1, so on change uevent on those devices, udev rules 68-del-part-nodes.rules will run /usr/bin/partx -d --nr 1-1024 $env{DEVNAME}. This makes for example disappear my boot partition (/dev/sda1), and so breaks my boot.

   I think size should be checked before making the map. Possible fix:

diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
index 9384de0..abdf933 100644
--- a/libmultipath/structs_vec.c
+++ b/libmultipath/structs_vec.c
@@ -300,6 +300,7 @@ int adopt_paths(vector pathvec, struct multipath *mpp,
 	int i, ret;
 	struct path * pp;
 	struct config *conf;
+	size_t count;

 	if (!mpp)
 		return 0;
@@ -307,6 +308,23 @@ int adopt_paths(vector pathvec, struct multipath *mpp,
 	if (update_mpp_paths(mpp, pathvec))
 		return 1;

+	count = 0;
+	vector_foreach_slot (pathvec, pp, i) {
+		if (!strncmp(mpp->wwid, pp->wwid, WWID_SIZE)) {
+			if (pp->size != 0 && mpp->size != 0 &&
+			    pp->size != mpp->size) {
+				condlog(3, "%s: size mismatch for %s, not adding path",
+					pp->dev, mpp->alias);
+				continue;
+			}
+		count += 1;
+		}
+	}
+	if (count < 2) {
+		condlog(3, "%s: only one path, cancel creation", mpp->alias);
+		return 1;
+	}
+
 	vector_foreach_slot (pathvec, pp, i) {
 		if (!strncmp(mpp->wwid, pp->wwid, WWID_SIZE)) {
 			if (pp->size != 0 && mpp->size != 0 &&

2. A more generic issue with UFS. What if my two LUN have the same size? multipath would consider them as the same! I do think UFS storages should be blacklisted from multipath. But I don´t know how to do that. Note that this issue could also happen with 0.11.1.

[mwilck]

Of course, all those disks have the same wwid (as it is the same storage). So multipath tries to handle them.

The term "of course" is wrong here. Different LUNs must not have the same WWID, period. This is misbehavior of your storage controller. In your specific case, comparing the size prevents the problem, but, as you wrote yourself, what if you had split your storage into multiple LUNs of the same size?

If you don't need multipath at all, just disable multipathd.service, or easier even, uninstall the multipath package.

For blacklisting, put the following in /etc/multipath.conf:

blacklist {
    wwid <THE WWID OF YOUR DISK GOES HERE>
}

That aside, what's your configuration setting for find_multipaths?
Also, please provide the output of udevadm info /dev/sdX for the different LUNs.

[fhost]

Of course, all those disks have the same wwid (as it is the same storage). So multipath tries to handle them.

The term "of course" is wrong here. Different LUNs must not have the same WWID, period. This is misbehavior of your storage controller. In your specific case, comparing the size prevents the problem, but, as you wrote yourself, what if you had split your storage into multiple LUNs of the same size?

Oh, I did not think of that. Good point...

Yet, I still think it is an issue to generate the multipath before being sure. But maybe my case has no way to happen (as wwids should not be the same) but then why checking the size?

If you don't need multipath at all, just disable multipathd.service, or easier even, uninstall the multipath package.

For blacklisting, put the following in /etc/multipath.conf:

blacklist {
    wwid <THE WWID OF YOUR DISK GOES HERE>
}

The issue is that it fails on first boot, so I cannot do that.

That aside, what's your configuration setting for find_multipaths? Also, please provide the output of udevadm info /dev/sdX for the different LUNs.

I don´t have any configuration. On show config, it says "no".

/dev/sda

P: /devices/platform/soc@0/1d84000.ufshc/host0/target0:0:0/0:0:0:0/block/sda
M: sda
J: b8:0
U: block
T: disk
D: b 8:0
N: sda
L: 0
S: disk/by-path/platform-1d84000.ufshc-scsi-0:0:0:0
S: disk/by-id/scsi-28ce38e0000000000
S: disk/by-diskseq/9
Q: 9
E: DEVPATH=/devices/platform/soc@0/1d84000.ufshc/host0/target0:0:0/0:0:0:0/block/sda
E: DEVNAME=/dev/sda
E: DEVTYPE=disk
E: DISKSEQ=9
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
E: USEC_INITIALIZED=10199964
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=KIOXIA
E: SCSI_VENDOR_ENC=KIOXIA\x20\x20
E: SCSI_MODEL=THGJFJT1E45BATPC
E: SCSI_MODEL_ENC=THGJFJT1E45BATPC
E: SCSI_REVISION=0300
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: SCSI_IDENT_SERIAL=94LEP3340R
E: SCSI_IDENT_LUN_EUI64=8ce38e0000000000
E: ID_VENDOR=KIOXIA
E: ID_VENDOR_ENC=KIOXIA\x20\x20
E: ID_MODEL=THGJFJT1E45BATPC
E: ID_MODEL_ENC=THGJFJT1E45BATPC
E: ID_REVISION=0300
E: ID_TYPE=disk
E: ID_BUS=scsi
E: ID_SERIAL=28ce38e0000000000
E: ID_SERIAL_SHORT=8ce38e0000000000
E: ID_SCSI_SERIAL=94LEP3340R
E: DM_MULTIPATH_DEVICE_PATH=0
E: ID_PATH=platform-1d84000.ufshc-scsi-0:0:0:0
E: ID_PATH_TAG=platform-1d84000_ufshc-scsi-0_0_0_0
E: ID_PART_TABLE_UUID=7007baf8-f0c0-49dc-8373-59e9464ec680
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_GPT_AUTO_ROOT_DISK=1
E: ID_DISSECT_IMAGE=1
E: ID_DISSECT_PART1_DESIGNATOR=esp
E: ID_FACTORY_RESET=unspecified
E: DEVLINKS=/dev/disk/by-path/platform-1d84000.ufshc-scsi-0:0:0:0 /dev/disk/by-id/scsi-28ce38e0000000000 /dev/disk/by-diskseq/9
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

/dev/sdb

P: /devices/platform/soc@0/1d84000.ufshc/host0/target0:0:0/0:0:0:1/block/sdb
M: sdb
J: b8:16
U: block
T: disk
D: b 8:16
N: sdb
L: 0
S: disk/by-diskseq/10
S: disk/by-path/platform-1d84000.ufshc-scsi-0:0:0:1
S: disk/by-id/scsi-28ce38e0000000000
Q: 10
E: DEVPATH=/devices/platform/soc@0/1d84000.ufshc/host0/target0:0:0/0:0:0:1/block/sdb
E: DEVNAME=/dev/sdb
E: DEVTYPE=disk
E: DISKSEQ=10
E: MAJOR=8
E: MINOR=16
E: SUBSYSTEM=block
E: USEC_INITIALIZED=10200192
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=KIOXIA
E: SCSI_VENDOR_ENC=KIOXIA\x20\x20
E: SCSI_MODEL=THGJFJT1E45BATPC
E: SCSI_MODEL_ENC=THGJFJT1E45BATPC
E: SCSI_REVISION=0300
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: SCSI_IDENT_SERIAL=94LEP3340R
E: SCSI_IDENT_LUN_EUI64=8ce38e0000000000
E: ID_VENDOR=KIOXIA
E: ID_VENDOR_ENC=KIOXIA\x20\x20
E: ID_MODEL=THGJFJT1E45BATPC
E: ID_MODEL_ENC=THGJFJT1E45BATPC
E: ID_REVISION=0300
E: ID_TYPE=disk
E: ID_BUS=scsi
E: ID_SERIAL=28ce38e0000000000
E: ID_SERIAL_SHORT=8ce38e0000000000
E: ID_SCSI_SERIAL=94LEP3340R
E: DM_MULTIPATH_DEVICE_PATH=0
E: ID_PATH=platform-1d84000.ufshc-scsi-0:0:0:1
E: ID_PATH_TAG=platform-1d84000_ufshc-scsi-0_0_0_1
E: ID_PART_TABLE_UUID=9ea2c5eb-9b3f-8482-4436-23e7670b50b3
E: ID_PART_TABLE_TYPE=gpt
E: DEVLINKS=/dev/disk/by-diskseq/10 /dev/disk/by-path/platform-1d84000.ufshc-scsi-0:0:0:1 /dev/disk/by-id/scsi-28ce38e0000000000
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

[mwilck]

Yet, I still think it is an issue to generate the multipath before being sure

Well, you have hit one of the hardest problems multipath-tools is facing: how to be sure? It's mostly consensus that we can / must rely on the WWID (actually, ID_SERIAL udev property) to assemble multipath LUNs.

The issue is that it fails on first boot, so I cannot do that.

If you can run show config, you should also be able to run remove map 28ce38e0000000000.

Or you could use the command line parameter nompath or multipath=off, which both disable device mapper multipath globally.

SCSI_IDENT_LUN_EUI64=8ce38e0000000000

So the controller sets the EUI64-based device ID (which we assume to be reliable) and uses the same value for both disks. That's ugly.

I don´t have any configuration. On show config, it says "no".

Strange. The upstream default is strict, with which this wouldn't have happened. no aka off is actually deprecated because it implies inconsistent behavior between multipath and multipathd (see multipath.conf(5)). This may be worth an Ubuntu bug report.

[fhost]

Yet, I still think it is an issue to generate the multipath before being sure

Well, you have hit one of the hardest problems multipath-tools is facing: how to be sure? It's mostly consensus that we can / must rely on the WWID (actually, ID_SERIAL udev property) to assemble multipath LUNs.

Yes, but then (i.e. following the validation via wwid), the size of the paths are checked, and if they mismatch, path is not added, and you end up with only one path in the multipath, and still make the multipath. This also makes multipathd returns that all paths (even the one not added) should be in the multipath (multipath -u <dev>) and add the wwid in the wwids file as a valid wwid, which is not really the case. For me this flow is not correct (independently of my issue with UFS).

The issue is that it fails on first boot, so I cannot do that.

If you can run show config, you should also be able to run remove map 28ce38e0000000000.

Or you could use the command line parameter nompath or multipath=off, which both disable device mapper multipath globally.

My purpose is to provide a "generic" Ubuntu image on some specific platforms. I don't want to disable multipath completely (my image needs to have the same behaviour as a generic Ubuntu one). And I can't be sure that the WWID will be the same on every platform. But I guess I won't have a choice other than listing all the possible WWIDs in the blacklist and pushing that configuration during image build.

SCSI_IDENT_LUN_EUI64=8ce38e0000000000

So the controller sets the EUI64-based device ID (which we assume to be reliable) and uses the same value for both disks. That's ugly.

I don´t have any configuration. On show config, it says "no".

Strange. The upstream default is strict, with which this wouldn't have happened. no aka off is actually deprecated because it implies inconsistent behavior between multipath and multipathd (see multipath.conf(5)). This may be worth an Ubuntu bug report.

Sorry! typo... it is "on", not "no"...
And that's the default on Ubuntu (patched in packaging). So I guess it is the behaviour they want.

[mwilck]

Yes, but then (i.e. following the validation via wwid), the size of the paths are checked, and if they mismatch, path is not added, and you end up with only one path in the multipath, and still make the multipath

IMO we're entering the realm of "undefined behavior" here.

I guess we could take a patch like what you proposed above, which would cover your concrete use case, but would not help if the LUNs on the controller had the same size. I think there is no general solution for the latter case. We'd need to think it through. I guess if a situation like this is encountered during startup, multipathd might consider the WWID in question as "failed" and never try to set it up again.

Feel free to send a patch to the dm-devel mailing list for review. I'll have no time to work on this in the near future.

Or you could use the command line parameter nompath or multipath=off, which both disable device mapper multipath globally.

My purpose is to provide a "generic" Ubuntu image on some specific platforms. I don't want to disable multipath completely (my image needs to have the same behaviour as a generic Ubuntu one).

You can create that generic image, and it will work. Users would just need to add multipath=off to the kernel command line. That's not asking a lot, and simpler than trying to figure out an automatic setup algorithm that works for arbitrarily broken systems.

Sorry! typo... it is "on", not "no"...

Ok, that makes sense.

[mwilck]

Sorry, closed by mistake.