FAILED checksums using sha256sum in windows 11

[wu0lss4j]

I've just noticed this error when checking hashes of image files (like .HEIC). Note: this error does not occur when checking hashes of text files.

Two ways to generate hashes:

1. sha256sum * > hash.256
2. sha256sum --tag * > hash.256 -- BSD style

The first yields hash *filename ← note the asterisk
The second yields SHA256 (filename) = hash ← filename between parenthesis

Regardless, when using sha256sum -c hash.256 will always yield FAILED

If I copy both the hashes files and the hash.256 to a linux install, it will check hashes just fine, albeit some formatting is necessary to the filename component.

[RenjiSann]

Does this only happens with several image files ?
Could you give us a corpus of files you can reproduce the errors on ?
I would understand that you don't want to share private photos.

[YDX-2147483647]

I meet the same issue for TTF files on Windows 10.
In my case, all affected files are binary, but not all binary files are affected.

It looks like if the file contains \r\n, then sha256sum --check (from coreutils) always reads it as a text file, no matter * exists or not and --binary is passed or not.
However, when generating hashes, sha256sum reads it as a binary file if appropriate.
As a result, they don't match.

Minimal example

Some shells may insert \n or \r\n when piping data, so I use python here to make it more reproducible.

from pathlib import Path
from subprocess import CompletedProcess
from subprocess import run as _run
from hashlib import sha256


def run(args: str, /, *, stdin: str | None = None) -> CompletedProcess[str]:
    return _run(args.split(), input=stdin, text=True, capture_output=True)


def assert_eq(
    result: CompletedProcess[str], /, *, exit_code=0, stdout="", stderr=""
) -> None:
    assert result.returncode == exit_code
    assert result.stdout == stdout
    assert result.stderr == stderr

# Prepare
file = b"\x00\x00\r\n\x00\x00"  # These are the bytes around offset 43134 of a real TTF file.
Path("a.fake-ttf").write_bytes(file)

hash_binary = "1c671d7322d49cd2726475f4b8a8b50f27b454789e23a31c6ac14014740d8e58"
hash_text = "73c0a22f7ed1c54ebde921712ced33d9458975cd86e9fe4757257fdd6293e512"

assert hash_binary == sha256(file).hexdigest()
assert hash_text == sha256(file.replace(b"\r\n", b"\n")).hexdigest()

# Calculate hashes
assert_eq(
    run("coreutils sha256sum a.fake-ttf"),
    stdout=f"{hash_binary} *a.fake-ttf\n",
)
assert_eq(
    run("coreutils sha256sum --binary a.fake-ttf"),
    stdout=f"{hash_binary} *a.fake-ttf\n",
)
assert_eq(
    run("coreutils sha256sum --text a.fake-ttf"),
    stdout=f"{hash_text}  a.fake-ttf\n",
)

# Check hashes
assert_eq(
    run(
        "coreutils sha256sum --check",
        stdin=f"{hash_text}  a.fake-ttf\n",
    ),
    stdout="a.fake-ttf: OK\n",
)
assert_eq(
    run(
        "coreutils sha256sum --text --check",
        stdin=f"{hash_text}  a.fake-ttf\n",
    ),
    stdout="a.fake-ttf: OK\n",
)
assert_eq(
    run(
        "coreutils sha256sum --check",
        stdin=f"{hash_binary} *a.fake-ttf\n",
    ),
    # 💥 This should not fail!
    exit_code=1,
    stdout="a.fake-ttf: FAILED\n",
    stderr="sha256sum: WARNING: 1 computed checksum did NOT match\n",
)
assert_eq(
    run(
        "coreutils sha256sum --binary --check",
        stdin=f"{hash_binary} *a.fake-ttf\n",
    ),
    # 💥 This should not fail!
    exit_code=1,
    stdout="a.fake-ttf: FAILED\n",
    stderr="sha256sum: WARNING: 1 computed checksum did NOT match\n",
)

$ coreutils --version
coreutils 0.5.0 (multi-call binary)

I've also tried the lastest commit (6dac4d4 as of writing), the only change is that coreutils sha256sum --text a.fake-ttf now returns hash_binary instead of hash_text. The results of coreutils sha256sum --check … have not been changed.

Relevant code

coreutils/src/uucore/src/lib/features/checksum/compute.rs

Lines 23 to 37 in 00f77cc

	/// Necessary options when computing a checksum. Historically, these options
	/// included a `binary` field to differentiate `--binary` and `--text` modes on
	/// windows. Since the support for this feature is approximate in GNU, and it's
	/// deprecated anyway, it was decided in #9168 to ignore the difference when
	/// computing the checksum.
	pub struct ChecksumComputeOptions {
	/// Which algorithm to use to compute the digest.
	pub algo_kind: SizedAlgoKind,
	/// Printing format to use for each checksum.
	pub output_format: OutputFormat,
	/// Whether to finish lines with '\n' or '\0'.
	pub line_ending: LineEnding,
	}

So it relates to #9168 and #9695.

[wu0lss4j]

Does this only happens with several image files ? Could you give us a corpus of files you can reproduce the errors on ? I would understand that you don't want to share private photos.

Hi, I will collect some examples and share. Thank you for your patience and sorry for the long delay in answering.

[wu0lss4j]

I meet the same issue for TTF files on Windows 10. In my case, all affected files are binary, but not all binary files are affected.

This is great work. I have dedicated more time that I wanted to hashing and this fact that not all sha256sum calculations were borne equal... Looking fwd to read more developments from your side.

[wu0lss4j]

Here is quick and dirty example, I just ran:

coreutils.exe sha256sum coreutils.exe > hash.256
coreutils.exe sha256sum -c hash.256

and it still fails (crying emoji)

Note: if I run a different sha256sum binary, say from a stable debbie trixie like so: $ sha256sum -c hash.256 it returns: coreutils.exe: OK

Edit: apologies for the 10^6 edits, the ! was \me in the embedded image posting...

[wu0lss4j]

So,... short update, still on v0.5.0 if you generate a checksum using -t (text mode) then the --check works

> coreutils sha256sum -t coreutils.exe > hash.256

> cat hash.256
091e2e9eff09351b6470fbf01044cd4a73772d0a31efd9023384def013972c27  coreutils.exe

> coreutils sha256sum -c hash.256
coreutils.exe: OK

but... the issue remains with the default -b (binary mode)

> coreutils sha256sum -c hash.256
coreutils.exe: OK

> coreutils sha256sum -b coreutils.exe > hash.256

> cat hash.256
5f9f016a73e6c8e7c6f2c38b0528d3f98d8f7e86d148a446036d0dfc72741920 *coreutils.exe

> coreutils sha256sum -c hash.256
coreutils.exe: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

I had a look at the code, but I don't do rust... so I shan't use coreutils v0.5.0 in the foreseeable future.

[RenjiSann]

Are you running coreutils in a CYGWIN environment ?

Can you compare uutils' output with GNU's ?

[YDX-2147483647]

I run coreutils in the native Windows 10, so GNU coreutils are not available.
However, I've provided the output of python hashlib.sha256 in #6655 (comment), which should explain most things.

• coreutils sha256sum a.fake-ttf gives hash_binary, but coreutils sha256sum --check a.fake-ttf only recognizes hash_text.
• Both PowerShell's Get-FileHash a.fake-ttf and Nushell open a.fake-ttf | hash sha256 also give hash_binary.
• If I send a.fake-ttf to Ubuntu 24.04 and run sha256sum a.fake-ttf, the output is also hash_binary.

As you can see, it's coreutils sha256sum --check that is problematic, and coreutils sha256sum is fine.

[wu0lss4j]

Are you running coreutils in a CYGWIN environment ?

and?

Can you compare uutils' output with GNU's ?

yes, debian trixie distro's own sha256sum binary hashes and verifies correctly coreutils.exe binary, including the hash as calculated by uutils-coreutils' windows binary

I'm no expert but maybe coreutils.exe's logic is buggy for the -c/--check switch