s3: --s3-no-head-object makes server side copy fail with "corrupted on transfer: sizes differ N vs 0" and deletes the destination

[anatoly314]

Before you start

• I have searched the forum and the existing issues for this problem.
• I have tested with the latest beta or stable release and can still reproduce the problem.
• This is a reproducible bug, not a usage question.

What is the problem you are having with rclone?

With --s3-no-head-object (or no_head_object = true in the remote config), every server-side copy on the S3 backend fails deterministically:

ERROR : file.txt: Failed to copy: corrupted on transfer: sizes differ src(...) 6 vs dst(...) 0

The copy itself succeeds on the server — the destination object is created, correct and complete — but rclone then declares it corrupt and deletes it ("Removing failed copy"), retries, and fails the same way. Because Move/rename on S3 is implemented as server-side copy + delete, this also breaks every rename, including renames through rclone mount (surfaces as EIO to the application). The source object is retained, so there is no data loss, but any workflow involving copy/move/rename within or between S3 remotes is broken while the flag is set.

The failure is purely client-side and provider-independent — reproduced on MinIO, and the mechanism (below) involves no server response at all, so it applies equally to AWS, B2 S3-compatible, etc.

Mechanism (verified in source on master, commit 9612668)

1. backend/s3/s3.go — (*Fs).Copy issues CopyObject, which succeeds. The CopyObject response does not carry the new object's size, so rclone builds a handle for the destination via dstObj, err := f.NewObject(ctx, remote).

2. NewObject → newObjectWithInfo with info == nil:

   } else if !o.fs.opt.NoHeadObject {
       err := o.readMetaData(ctx) // reads info and meta, returning an error

   With NoHeadObject set, readMetaData is skipped entirely — no request of any kind is sent. o.bytes is never populated and stays at its Go zero value, 0. There is no "unknown size" sentinel (-1) on this path, so "never asked" is indistinguishable from "genuinely 0 bytes".

3. fs/operations/copy.go — the post-copy verification compares sizes: src 6 vs dst 0 → "corrupted on transfer" → the healthy destination object is deleted and the copy retried, failing identically.

The -vv log below shows it directly: size = 6 for src, size = 0 for dst, and Server Side Copies: 1 @ 0 B for a copy that succeeded server-side.

Note the asymmetry that makes this a bug rather than a documented trade-off: on the download path the flag is compensated for (metadata comes from the GET response headers / listings), but on the server-side copy path nothing compensates — the size check runs against a value that was never fetched.

Reproduction (MinIO, fully scripted)

docker network create repro

docker run -d --name minio --network repro \
    -e MINIO_ROOT_USER=admin -e MINIO_ROOT_PASSWORD=admin12345 \
    minio/minio:RELEASE.2025-09-07T16-13-09Z server /data

alias rc='docker run -i --rm --network repro \
    -e RCLONE_CONFIG_T_TYPE=s3 \
    -e RCLONE_CONFIG_T_PROVIDER=Minio \
    -e RCLONE_CONFIG_T_ENDPOINT=http://minio:9000 \
    -e RCLONE_CONFIG_T_ACCESS_KEY_ID=admin \
    -e RCLONE_CONFIG_T_SECRET_ACCESS_KEY=admin12345 \
    rclone/rclone:1.74.4'

# seed
rc mkdir T:bucket
echo hello | rc rcat T:bucket/src/file.txt

# control: server-side copy WITHOUT the flag — works
rc copy -v T:bucket/src T:bucket/dst
#   ... Copied (server-side copy) ... Server Side Copies: 1 @ 6 B

# with the flag — fails, and the (good) destination object is deleted
rc copy -v --s3-no-head-object T:bucket/src T:bucket/dst2
#   ERROR : file.txt: Failed to copy: corrupted on transfer: sizes differ src(...) 6 vs dst(...) 0
rc ls T:bucket/dst2   # empty

Suggested directions for a fix (any one would do)

• After a server-side copy, populate the destination object's size from the source object (the sizes are equal by definition of CopyObject), or
• On the Copy path specifically, perform readMetaData even when no_head_object is set (one HEAD per server-side copy is negligible next to the copy itself), or
• Treat the never-fetched size as unknown (-1) so the size check is skipped, or
• At minimum, document that no_head_object is incompatible with server-side copy/move.

rclone version

rclone v1.74.4
- os/version: alpine 3.24.1 (64 bit)
- os/kernel: 6.12.72-linuxkit (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.26.5
- go/linking: static
- go/tags: none

Also verified in source on master (commit 9612668) — the code path is unchanged. Reproduced on linux/amd64 and darwin/arm64 as well.

Operating system

Linux (docker rclone/rclone:1.74.4), also macOS

Which cloud storage system are you using?

S3 (reproduced with provider Minio; mechanism is client-side so provider-independent — also hit in production against Backblaze B2's S3-compatible API)

Output of rclone config redacted

Config is passed via environment variables (see repro script); the equivalent ini is:

[T]
type = s3
provider = Minio
endpoint = http://minio:9000
access_key_id = XXX
secret_access_key = XXX

The command you were trying to run

rclone copy -vv --retries 1 --s3-no-head-object T:bucket/src T:bucket/dst2

A log from the command with the -vv flag

2026/07/19 10:38:42 DEBUG : rclone: Version "v1.74.4" starting with parameters ["rclone" "copy" "-vv" "--retries" "1" "--s3-no-head-object" "T:bucket/src" "T:bucket/dst2"]
2026/07/19 10:38:42 DEBUG : Creating backend with remote "T:bucket/src"
2026/07/19 10:38:42 NOTICE: Config file "/config/rclone/rclone.conf" not found - using defaults
2026/07/19 10:38:42 DEBUG : Setting type="s3" for "T" from environment variable RCLONE_CONFIG_T_TYPE
2026/07/19 10:38:42 DEBUG : Setting provider="Minio" for "T" from environment variable RCLONE_CONFIG_T_PROVIDER
2026/07/19 10:38:42 DEBUG : Setting endpoint="http://minio-nho:9000" for "T" from environment variable RCLONE_CONFIG_T_ENDPOINT
2026/07/19 10:38:42 DEBUG : fs cache: renaming cache item "T:bucket/src" to be canonical "T{ckuwj}:bucket/src"
2026/07/19 10:38:42 DEBUG : Creating backend with remote "T:bucket/dst2"
2026/07/19 10:38:42 DEBUG : fs cache: renaming cache item "T:bucket/dst2" to be canonical "T{ckuwj}:bucket/dst2"
2026/07/19 10:38:42 DEBUG : file.txt: Need to transfer - File not found at Destination
2026/07/19 10:38:42 DEBUG : S3 bucket bucket path dst2: Waiting for checks to finish
2026/07/19 10:38:42 DEBUG : S3 bucket bucket path dst2: Waiting for transfers to finish
2026/07/19 10:38:42 DEBUG : file.txt: size = 6 (S3 bucket bucket path src)
2026/07/19 10:38:42 DEBUG : file.txt: size = 0 (S3 bucket bucket path dst2)
2026/07/19 10:38:42 ERROR : file.txt: corrupted on transfer: sizes differ src(S3 bucket bucket path src) 6 vs dst(S3 bucket bucket path dst2) 0
2026/07/19 10:38:42 INFO  : file.txt: Removing failed copy
2026/07/19 10:38:42 ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: sizes differ src(S3 bucket bucket path src) 6 vs dst(S3 bucket bucket path dst2) 0
2026/07/19 10:38:42 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 0 / 0, -, Listed 1
Server Side Copies:     1 @ 0 B
Elapsed time:         0.0s

2026/07/19 10:38:42 DEBUG : 9 go routines active
2026/07/19 10:38:42 NOTICE: Failed to copy: corrupted on transfer: sizes differ src(S3 bucket bucket path src) 6 vs dst(S3 bucket bucket path dst2) 0

(Repeated Setting ... from environment variable lines trimmed for readability; nothing else removed.)

Anything else?

How we hit it in production: an rclone FUSE mount of a Backblaze B2 prefix (S3-compatible API, restricted per-user keys). We set --s3-no-head-object to work around B2 returning 403 (not 404) for HEAD on nonexistent objects (same trigger as #8142), which confuses the mount-root file-vs-directory probe. The flag fixed the probe but broke every delete in the application above the mount (delete = rename into a trash dir = server-side copy). Our workaround: drop the flag and slash-terminate the mount root (bucket/prefix/), which skips the probe instead — mentioned in case it helps others who reached for no_head_object for the same reason.

Searched for duplicates: no existing issue covers this. #8500 shows the same error string but is an upload through a mangling proxy; the forum thread copy-to-scality-s3-corrupted-on-transfer-sizes-differ-xxx-vs-0 is a server-side, Scality-specific problem. The docs for no_head_object ("If set, do not do HEAD before GET when getting objects") give no hint that server-side copy stops working.

[ncw]

directions for a fix (any one would do)

• After a server-side copy, populate the destination object's size from the source object (the sizes are equal by definition of CopyObject), or

This is what we've done elsewhere in the code. The population of data should only be done on no_head_objecy