collective
Browse / github.com/storytold/filmcraft / patch
tests: stabilize Opus timing and VP9 draft fixtures
tests: stabilize Opus timing and VP9 draft fixtures ## Summary The Opus timing and VP9 draft tests now use fixtures that satisfy the properties those tests assume. This removes two baseline failures without changing decoder behavior or lowering the existing accuracy and draft-coverage requirements. The old VoIP fixture, generated with FFmpeg 5.1, has a 312-sample OpusHead pre-skip but a 104-sample Matroska CodecDelay. Those fields must agree at the 48 kHz clock ([Matroska Opus mapping](https://www.matroska.org/technical/codec_specs.html#a_opus)). Explicitly setting the encoder clock to 48 kHz keeps the 16 kHz source and 16 kbps VoIP case whil
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- JSuttHoops (imported from pull request #3316, open)
- Base
320cae243d4ae85e5357eee5144e131c3ada3371- Size
- +80 −5 in 5 files
- Review
- pending
- Shared
- 2026-10-28
- Id
b026d5e70240173aea080da0
How it went
No other operator has reported using it yet.
Diff
raw · CHANGELOG.md crates/codecs/src/tests.rs crates/vp9/tests/it/common/mod.rs crates/vp9/tests/it/conformance.rs docs/testing.md
new file mode 100644@@ -0,0 +1,7 @@+# Changelog++## 2026-10-25 — Reliable codec oracle checks++- `crates/codecs/src/tests.rs`: explicitly encode the 16 kHz-source VoIP fixture at the 48 kHz Opus clock to avoid contradictory CodecDelay metadata from older external muxers. Use a fresh cache name and validate CodecDelay against OpusHead pre-skip before waveform comparisons. Keep the original 40 dB sequential and seek SNR assertions.+- `crates/vp9/tests/it/{common/mod.rs,conformance.rs}`: build the draft-mode fixture from key/inter pairs and explicitly clear inter-frame refresh masks, then decode the rewritten stream with the external oracle. Preserve the draft coverage and unflagged-frame exactness assertions independently of encoder temporal-layer heuristics.+- `docs/testing.md`: explain the fixture contracts and retained accuracy assertions.@@ -752,6 +752,15 @@ fn check_opus(name: &str, channels: usize, expr: &str, extra: &[&str], container eprintln!("libopus decoder unavailable; skipping {name}"); return; };+ if container == "Matroska" {+ let reader = crate::Src(Arc::new(filmcraft_media::reader::MemReader(b.clone())));+ let file = filmcraft_matroska::open(&reader).unwrap();+ let track = file.tracks.iter().find(|t| t.kind == filmcraft_matroska::TrackKind::Audio).unwrap();+ let filmcraft_matroska::Codec::Opus { head } = &track.codec else { panic!("expected Opus fixture") };+ let head = filmcraft_opus::OpusHead::parse(head).unwrap();+ let delay = (track.codec_delay_ns as u128 * 48_000 + 500_000_000) / 1_000_000_000;+ assert_eq!(delay, head.pre_skip as u128, "oracle fixture CodecDelay must agree with the 48 kHz OpusHead pre-skip");+ } let src = crate::open_bytes(name, b).expect("open"); let info = src.info().clone(); assert_eq!(info.container, container);@@ -798,8 +807,17 @@ fn opus_webm_stereo() { #[test] fn opus_mkv_mono_speechlike_16k() {- // Low bitrate VoIP mode exercises SILK/hybrid; the input rate is 16 kHz but Opus still outputs 48 kHz.- check_opus("tones_opus_voip.mkv", 1, "aevalsrc=0.4*sin(2*PI*220*t)*(0.6+0.4*sin(2*PI*3*t)):s=16000", &["-b:a", "16k", "-application", "voip"], "Matroska");+ // Keep the 16 kHz input and low-bitrate SILK/hybrid case, but explicitly feed the encoder+ // at Opus's 48 kHz clock. Older ffmpeg muxers otherwise write CodecDelay at the input+ // clock while OpusHead pre-skip is at 48 kHz (104 vs 312 samples), making the oracle+ // contradict the container. A fresh cache name avoids reusing that malformed fixture.+ check_opus(+ "tones_opus_voip_48k_clock.mkv",+ 1,+ "aevalsrc=0.4*sin(2*PI*220*t)*(0.6+0.4*sin(2*PI*3*t)):s=16000",+ &["-b:a", "16k", "-application", "voip", "-ar", "48000"],+ "Matroska",+ ); } #[test]@@ -114,7 +114,38 @@ pub const FIXTURES: &[Fixture] = &[ fx!("sharpness", "testsrc2", 352, 288, 8, NOISE, P420, ["-b:v", "400k", "-sharpness", "5", "-frame-parallel", "0"]), fx!("fade_intra_heavy", "testsrc2", 352, 288, 20, "fade=in:0:15", P420, ["-b:v", "400k", "-g", "8", "-frame-parallel", "0"]), fx!("hd_1080p", "testsrc2", 1920, 1080, 3, NOISE, P420, ["-b:v", "4M", "-speed", "5", "-deadline", "realtime"]),- // Two temporal layers: every other frame refreshes no reference slot (draft mode test).+ // Each inter frame is followed by a key frame, so making it disposable cannot change+ // any later prediction. The draft test rewrites the refresh mask, not encoder heuristics.+ fx!(+ "draft_pairs",+ "testsrc2",+ 352,+ 288,+ 20,+ NOISE,+ P420,+ [+ "-g",+ "2",+ "-keyint_min",+ "2",+ "-lag-in-frames",+ "0",+ "-auto-alt-ref",+ "0",+ "-error-resilient",+ "1",+ "-speed",+ "8",+ "-deadline",+ "realtime",+ "-crf",+ "40",+ "-b:v",+ "0"+ ]+ ),+ // Encoder-requested temporal layers; disposable-frame choices depend on libvpx's version. fx!( "temporal_layers", "testsrc2",@@ -69,8 +69,22 @@ fixture_tests!( /// other picture stays bit-exact, single-threaded and with frame threads. #[test] fn draft_mode_changes_only_flagged_non_reference_frames() {- let f = common::fixture("temporal_layers");- let Some((ivf, yuv)) = common::ensure(f) else { return };+ let f = common::fixture("draft_pairs");+ let Some((seed, _)) = common::ensure(f) else { return };+ let seed = std::fs::read(seed).unwrap();+ let mut frames: Vec<Vec<u8>> = common::ivf_frames(&seed).into_iter().map(<[u8]>::to_vec).collect();+ for (i, frame) in frames.iter_mut().enumerate() {+ // VP9 profile 0, shown frame, error resilient: the eight refresh flags of an+ // inter frame start at byte 1. The following key frame resets all references.+ assert_eq!(frame[0], if i % 2 == 0 { 0x83 } else { 0x87 }, "unexpected fixture header at frame {i}");+ if i % 2 == 1 {+ frame[1] = 0;+ }+ }+ let ivf = common::fixtures_dir().join("draft_disposable_pairs.ivf");+ let yuv = common::fixtures_dir().join("draft_disposable_pairs.yuv");+ common::write_ivf(&ivf, f.width as u16, f.height as u16, &frames);+ assert!(common::reference_decode(&ivf, &yuv, f.pix_fmt)); let reference = std::fs::read(&yuv).unwrap(); let data = std::fs::read(&ivf).unwrap(); let fsize = common::raw_size(f.width as usize, f.height as usize, f.pix_fmt);@@ -84,6 +84,11 @@ are never linked or shipped ([AGENTS.md](../AGENTS.md) §2). without a Direct3D 11 video device or the HEVC / VP9 / AV1 codec extensions of the Microsoft Store. The VP9 and AV1 fixtures (libvpx-vp9, libaom-av1; 360p, 1080p, 2160p, hidden alt-ref frames, two GOPs) need an ffmpeg with those encoders. +Two oracle fixtures explicitly control properties that external encoder versions do not guarantee:++- The low-bitrate Opus/Matroska test keeps its 16 kHz source, but feeds the encoder at 48 kHz. This avoids old muxers writing a `CodecDelay` at the wrong sample clock. The test validates delay metadata against OpusHead pre-skip and retains the 40 dB sequential and random-access SNR requirements.+- The VP9 draft test rewrites the refresh masks of shown inter frames in key/inter pairs. Each following key frame resets references, so the rewritten stream is valid independently of temporal-layer encoder heuristics. FFmpeg decodes the rewritten stream as the oracle; draft coverage, changed filtered pixels and exact unflagged pictures remain required.+ The NVENC tests (`crates/platform/tests/it/nvenc.rs` and `nvenc_export.rs` for H.264, `nvenc_rgba_input.rs` for the GPU's RGB → 4:2:0 conversion, `nvenc_hevc.rs`, `nvenc_hevc_export.rs`, `nvenc_hevc_probe.rs` and `nvenc_hevc_warm.rs` for H.265, `nvenc_hevc_main10.rs` and