ptx: performance improvement - avoid per-word filename clones and lin…

[Aisha630]

ptx: fix super-linear slowdown on long lines, and cut peak memory usage

Summary

While evaluating uutils coreutils as part of a research project, I ran a fuzzing campaign and noticed that ptx would appear to hang on large inputs. By "hang," I mean it kept running for several minutes and, on bigger files, effectively never finished, even within an hour. After reproducing it and reading the source, I found it was not an infinite loop but an algorithmic problem. ptx was repeatedly rescanning the same line prefixes.

This PR fixes that. On a pathological 2 MB single-line input, ptx goes from ~10 s to ~0.34 s (~29× faster), and it also uses ~19% less peak memory than the original.

How I found it

My corpus was valid UTF-8 with very few or no newlines. That is the worst case for ptx. With no newlines, the whole file is one giant "line."

Doubling the line length doubled the runtime while the file stayed 2 MB, the signature of Θ(n·L) work, which degenerates to Θ(n²) for a single line.

Root cause

Word positions are found as byte offsets, but output indexes into a Vec<char>, so ptx converts byte → character offsets in prepare_line_chunks:

// old: recomputed from the start of the line for every word occurrence
let ref_char_position = line[..word_ref.position].chars().count();

line[..word_ref.position].chars().count() walks the line from the start every time. Since words later in a line sit at larger offsets, the conversion cost grows. ptx deliberately builds a Vec<char> for O(1) indexing, but this prefix rescan defeated that intent.

Fix, part 1 - eliminate the rescans (big speedup)

create_word_set already walks each line's matches in increasing byte order via find_iter, so I convert offsets with a running cursor (only counting the characters between the previous keyword and the current one) and cache the result as WordRef::char_position. ASCII lines get a fast path where byte == char, so the conversion is a no-op. prepare_line_chunks then just reads the cached value.

This collapses the curve from quadratic to flat:

xychart-beta
    title "Runtime vs line length, fixed 2 MB input (lower is better)"
    x-axis ["6KB", "25KB", "100KB", "400KB", "800KB", "1.6MB", "2MB(1 line)"]
    y-axis "Seconds" 0 --> 11
    line [0.39, 0.47, 0.82, 2.20, 3.76, 6.72, 10.02]
    line [0.33, 0.32, 0.32, 0.33, 0.35, 0.32, 0.34]

Loading

Upper curve = baseline (blows up with line length); flat lower curve = after the fix.

But this honestly made things worse in one dimension: caching a usize per word occurrence pushed average peak memory up ~16% (≈66.6 → ≈77.3 MiB). I wasn't happy shipping a memory regression, so I kept going.

Fix, part 2 - pay back the memory

I noticed that each WordRef also stored a cloned OsString filename, one owned copy per word occurrence. I replaced it with a file_idx: usize index into the existing FileMap, looking the name up only when emitting output. As a bonus, this removed an O(files) filename scan that ran on every output line in write_traditional_output; file_idx also disambiguates duplicate file arguments (ptx file file) directly, which the name-based scan had to handle in more complex way.

That more than pays for the cached offset. Final memory lands below the original baseline:

xychart-beta
    title "Average peak memory, 2 MB inputs (lower is better)"
    x-axis ["Baseline", "Rescan fix only", "Rescan fix + filename dedup"]
    y-axis "MiB" 0 --> 90
    bar [66.6, 77.3, 54.0]

Loading

Combined result

xychart-beta
    title "Worst case: single 2 MB line (lower is better)"
    x-axis ["Baseline", "Rescan fix", "Rescan fix + dedup"]
    y-axis "Seconds" 0 --> 11
    bar [10.02, 0.39, 0.34]

Loading

Metric (2 MB, single line)	Baseline	Rescan fix	Rescan fix + dedup
Runtime	10.02 s	0.39 s	0.34 s (~29× faster)
Peak memory	69.6 MiB	79.9 MiB	56.6 MiB
Avg peak memory (full sweep)	66.6 MiB	77.3 MiB	54.0 MiB (−19%)

Complexity goes from Θ(n·L) (→ Θ(n²) for one long line) to effectively Θ(n).

Correctness

This is a pure performance change with no behavioral difference:

• cargo test -p coreutils --test tests -- test_ptx - 44 passed, 0 failed.
• GNU tests/ptx/ptx.pl (coreutils 9.11) run against this build - all cases pass, apart from the pre-exisitng regex failure.

Shipped as a single, self-contained commit touching only src/uu/ptx/src/ptx.rs.

[sylvestre]

note: long comments (probably written by llm) aren't helpful

what we care about it benchmarks in tree (using divan) that demonstrate the improvement of your work
please add one in a different pr
thanks

[sylvestre]

why such long comments ?

[sylvestre]

same, long comments aren't super helpful

[Aisha630]

note: long comments (probably written by llm) aren't helpful

what we care about it benchmarks in tree (using divan) that demonstrate the improvement of your work please add one in a different pr thanks

noted. will make a new PR addressing these

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/date/date-locale-hour (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/pid-pipe (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/cut/bounded-memory (passes in this run but fails in the 'main' branch)