Ported the two annotation fixes, updated codec spec#16
Conversation
|
Hey Patrick — congrats, the Sony additions are a huge amount of ground covered. Before #16 lands: I ran it against our production corpus (316 real Olympus files,
It's the byte-1 anchor. Every file that breaks has blocks where the anchor Unrelated to your PR: master panics on 3 of our real DSS-SP files — Samples available for any of it. |
|
Yep, I hit that overrun in my ffmpeg code and fixed it, but never thought to loop back around to check if the rust code had the same issue. AI seems to think my samples are facing the problem at one end, and yours are the other end. That seems to check out, if your samples have lots of stops due to doctor/... pausing often. Most of my clips are just stright runs without pauses, only one with the pauses in it. |
The block-quantized QP walk from the same upstream work (hirparak#16) is deliberately omitted: it garbled 22 of 316 production Olympus recordings at voice-activated pause/re-sync blocks (fc=0/fc=19), where the byte-1 anchor is authoritative, not the frame count. Revisit once more split-file samples are available. - codec/ds2_qp: a SpeechExec-split file cut mid-frame yields a fixed-codebook index past C(64,11); mirror the reference decoder (drop pulse slot 0, place slots 1-10 at positions 9..0). - demux/dss, demux/ds2 (SP): total_frames counts every declared frame but the assembled stream drops compact/empty padding, so the frame walk can overrun the stream and panic in release builds (slice underflow at dss.rs:156 on real DSS-SP files). Clamp both slice ends and emit zero-padded trailing packets; regression tests for both paths. - spec: document QP7, the comb fallback, per-segment de-emphasis, and the anchor-based segmented demux.
|
Patch adjusted to pull that out. |
|
I found my issue, my golden masters I made for the annotations stuff got corrupted along the way and I did not notice, likely overwrote them by accident from a test decode. Getting sloppy here I guess, was too deep into the msv/dvf stuff. |
An SP block can declare more frames than the assembled stream holds (total_frames counts every declared frame, but compact/empty-block padding is dropped), so the final frame reads ran past the end of the stream and panicked in release builds (slice underflow at dss.rs:156 on real DSS-SP recordings). Clamp the reads and emit zero-padded trailing packets instead; regression tests for both demuxers. Also decode a fixed-codebook index at or past C(64,11) (possible only on corrupt or misframed input) with a deterministic pulse fallback: drop slot 0, place slots 1-10 at positions 9..0. Update CODEC_SPECIFICATION.md: QP7 decoder, per-segment de-emphasis, anchor-based segmented demuxing.
2 participants
src/demux/ds2.rs — replaced the anchor-based segmentation with the block-quantized walk (mirrors the FFmpeg ds2_qp_walk_blocks logic: min(fc, geometric-fit cap) frames per block, straddle-aware, dead-tail skipping, fc==0 = pause + reset). The byte-1 anchor is no longer consulted on linear reads. src/codec/ds2_qp.rs — added the C(64,11) comb-index overflow rule at the decode_combinatorial_index call site (drop slot 0, place slots 1-10 at positions 9..0). Verification:
file before after
annotation (magic 1) rejected → then garbage 82/82 frames ≤ 1 LSB main_text (magic 3) garbage (maxdiff 62474) 101/101 frames ≤ 1 LSB clean QP bit-exact bit-exact (unchanged)
broken (QP7 golden) golden byte-identical (untouched)
The Rust decoder now matches the NCH golden with the exact same diff counts as FFmpeg (nd 10578 / 12908). All 39 unit tests pass.
Worth noting: the demux fix alone (before adding comb) only recovered the aligned prefix (main_text 18/101), both fixes are genuinely required, which is why I flagged it before adding the comb rule.
Updated CODEC_SPECIFICATION.md