Skip to content

dd: keep short reads from corrupting re-blocked output#13459

Open
relative23 wants to merge 1 commit into
uutils:mainfrom
relative23:dd-partial-read-gathering
Open

dd: keep short reads from corrupting re-blocked output#13459
relative23 wants to merge 1 commit into
uutils:mainfrom
relative23:dd-partial-read-gathering

Conversation

@relative23

Copy link
Copy Markdown

Fixes #13458.

Two related defects made short reads (pipes, FIFOs, sockets — no iflag=fullblock) corrupt or mis-block the output when obs > ibs:

  1. fill_consecutive kept reading after a short read, so the next read landed at the following ibs-aligned chunk of the copy-loop buffer. The gap in between held stale buffer bytes, and the final truncate(bytes_total) kept that gap in the output while cutting the same number of real bytes off the end — silent corruption, since the record counts still match GNU (reproducer and analysis in dd: a short read corrupts the output when obs > ibs (stale filler byte written, real bytes dropped) #13458). The fix ends the fill at the first short read: the bytes received so far form one partial record, which is how GNU dd processes it. Complete ibs reads continue to fill the buffer exactly as before, so regular-file behavior is unchanged.

  2. Fixing that exposed a latent defect in BufferedOutput::write_blocks: when the pending partial block plus the incoming bytes still do not fill one obs block, the split index saturates to zero and the pending bytes were handed to the inner writer as a premature partial write (visible as 0+2 records out where GNU reports 0+1). They are now buffered until a block completes or the stream ends. This path was practically unreachable before this change, because sub-obs fills used to happen only once, at EOF.

Testing

  • New FIFO regression test (ibs=3 obs=6, two deliberately short reads): fails on main with ab\xDDc (stale byte injected, d dropped), passes with abcd after the fix; the stats 0+2 records in / 0+1 records out match GNU dd (verified against GNU on Debian stable). Ran 10× in a loop to guard against timing flakiness.
  • New unit test for BufferedOutput: two writes that together still do not fill one block are both buffered instead of being flushed prematurely.
  • cargo test -p uu_dd (89 tests) and the dd integration suite (127 passed, including the existing FIFO partial-read tests with ibs == obs, which are unaffected) pass; fmt, clippy and cspell are clean.

Note

The changed lines in fill_consecutive overlap textually with the refactor in #13373; the conflict is trivial and I am happy to rebase either PR once the other lands.

Comment thread tests/by-util/test_dd.rs Outdated
unix,
not(target_os = "macos"),
not(target_os = "freebsd"),
feature = "printf"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

printf is shell's built-in.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. This test drives the writer through sh -c, where printf is a shell builtin, not the uutils printf utility, so the feature dependency was unnecessary.

I removed feature = "printf" from the regression test and widened the cfgs of the two shared imports it uses (get_tests_binary and std::process::Command) by their actual use, so a dd-only build still compiles. Verified that the test is now discovered and passes with --no-default-features --features dd (it was silently compiled out there before), and that dd,printf, feat_os_unix and the default feature set are unchanged.

The two neighbouring FIFO tests carry the same historical gate even though they also only use the shell builtin. I left them untouched to keep this PR focused on the correctness fix; they are the same kind of test-configuration cleanup and can follow separately.

@relative23
relative23 force-pushed the dd-partial-read-gathering branch from b9bd209 to 7193662 Compare July 20, 2026 10:01
@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 3.04%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 338 untouched benchmarks
⏩ 46 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation du_max_depth_balanced_tree[(6, 4, 10)] 25.9 ms 25.1 ms +3.04%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing relative23:dd-partial-read-gathering (8199a64) with main (50b8ead)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/rm/isatty (fails in this run but passes in the 'main' branch)

Two related defects made short reads (pipes, FIFOs, sockets, no
iflag=fullblock) corrupt or mis-block the output when obs > ibs:

fill_consecutive kept reading after a short read, so the next read
landed at the following ibs-aligned chunk of the copy-loop buffer. The
gap in between held stale buffer bytes, and the final
truncate(bytes_total) kept that gap in the output while cutting the
same number of real bytes off the end -- silent corruption, since the
record counts still matched GNU. End the fill at the first short read
instead: the bytes received so far form one partial record, as in GNU
dd.

That exposed a latent defect in BufferedOutput::write_blocks: when the
pending partial block plus the incoming bytes still do not fill one obs
block, the split index saturated to zero and the pending bytes were
handed to the inner writer as a premature partial write. Buffer them
instead until a block completes or the final flush.

The regression test drives the writer through `sh -c`, where printf is
a shell builtin, so it does not need the printf utility feature; only
the shared imports it uses were gated on it. Widen those gates by use
instead, so the test also runs in a dd-only build.

Fixes uutils#13458.
@relative23
relative23 force-pushed the dd-partial-read-gathering branch from 7193662 to 8199a64 Compare July 20, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dd: a short read corrupts the output when obs > ibs (stale filler byte written, real bytes dropped)

2 participants