collective
Browse / github.com/storytold/filmcraft / patch
platform: read VA-API pictures through vaDeriveImage
platform: read VA-API pictures through vaDeriveImage Sibling PRs of this series add rows to the same revision-history tables in `docs/`; whichever merges second has a one-line conflict there (keep both rows). ## What Every decoded picture was copied out of the surface with vaGetImage into a second image before it was mapped - most of the readback's cost at 4K. vaDeriveImage maps the surface's own memory instead. - The session reads through vaDeriveImage first; the first failure (a driver that cannot derive this surface) logs once and switches the session back to vaGetImage. vaDeriveImage is loaded optionally (sym_opt!, new): an older libva ke
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- zsoli69 (imported from pull request #3214, open)
- Base
10d626ccc4888e0f020f9ca314c2bc9148ded4f4- Size
- +123 −22 in 9 files
- Review
- pending
- Shared
- 2026-10-28
- Id
70616b5b0fb5fb043efc2276
How it went
No other operator has reported using it yet.
Diff
raw · crates/platform/README.md crates/platform/src/biplanar.rs crates/platform/src/media_foundation/gpu.rs crates/platform/src/nvdec/cuvid.rs crates/platform/src/vaapi/ffi.rs crates/platform/src/vaapi/tests.rs crates/platform/src/vaapi/va.rs docs/hardware-parity.md docs/performance.md
@@ -105,8 +105,9 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox") bit-exact. Each picture is sent as one picture-parameter buffer, the scaling matrices, and a slice parameter + slice data buffer per slice (the NAL unit as stored, emulation prevention included). HEVC pictures are sent the same way per slice segment (scaling lists only when enabled, converted- to raster order). Pictures the DPB outputs are read back right away (`vaGetImage` into an NV12 or- P010 image, then `biplanar.rs`), **the one GPU to CPU copy**. Surfaces: the stream's DPB size plus two.+ to raster order). Pictures the DPB outputs are read back right away (`vaDeriveImage`, which maps the+ surface's own NV12 or P010 memory, or `vaGetImage` into an image where a driver cannot derive it;+ then `biplanar.rs`), **the one GPU to CPU copy**. Surfaces: the stream's DPB size plus two. - *frame_num gaps* (and every start at a non-IDR picture: an open-GOP seek) get "non-existing" frames like the software decoder's: copies of the latest decoded frame, mid-gray when there is none (`vaPutImage`). Pictures predicted from them (the leading pictures of an open GOP after a@@ -9,11 +9,14 @@ use std::sync::Arc; use filmcraft_frame::{Chroma, PixelData, VideoFrame, pool}; -/// A mapped biplanar surface: a luma plane followed by interleaved chroma, `stride` bytes per row.+/// A mapped biplanar surface: a luma plane (`stride` bytes per row) and interleaved chroma+/// (`chroma_stride` bytes per row; the same as `stride` for most surfaces, but a derived VA-API+/// image may pitch its planes differently). pub struct Biplanar<'a> { pub luma: &'a [u8], pub chroma: &'a [u8], pub stride: usize,+ pub chroma_stride: usize, /// Rows in the luma plane (the surface height, which may exceed the picture's). pub rows: usize, }@@ -48,8 +51,12 @@ pub fn to_frame(src: &Biplanar, g: &Geometry) -> Result<VideoFrame, String> { let (cox, coy) = (ox / 2, oy / 2); let needs_rows = oy.checked_add(h).ok_or("picture size overflows")?; let needs_cols = ox.checked_add(w).and_then(|c| c.checked_mul(bps)).ok_or("picture size overflows")?;- if src.rows < needs_rows || src.stride < needs_cols || src.stride < (cox + cw) * 2 * bps {- return Err(format!("decoded surface ({} rows, {} byte rows) is smaller than the {w}x{h} picture at {ox},{oy}", src.rows, src.stride));+ let needs_chroma_cols = cox.checked_add(cw).and_then(|c| c.checked_mul(2 * bps)).ok_or("picture size overflows")?;+ if src.rows < needs_rows || src.stride < needs_cols || src.chroma_stride < needs_chroma_cols {+ return Err(format!(+ "decoded surface ({} rows, {} / {} byte rows) is smaller than the {w}x{h} picture at {ox},{oy}",+ src.rows, src.stride, src.chroma_stride+ )); } let data = if bps == 1 { let mut yp = pool::take_u8(w * h);@@ -59,7 +66,7 @@ pub fn to_frame(src: &Biplanar, g: &Geometry) -> Result<VideoFrame, String> { let (mut u, mut v) = (pool::take_u8(cw * ch), pool::take_u8(cw * ch)); for y in 0..ch { // two passes over the row: each is a simple strided copy the compiler vectorises- let pairs = row(src.chroma, src.stride, coy + y, cox * 2, cw * 2)?.as_chunks::<2>().0;+ let pairs = row(src.chroma, src.chroma_stride, coy + y, cox * 2, cw * 2)?.as_chunks::<2>().0; u.extend(pairs.iter().map(|p| p[0])); v.extend(pairs.iter().map(|p| p[1])); }@@ -72,7 +79,7 @@ pub fn to_frame(src: &Biplanar, g: &Geometry) -> Result<VideoFrame, String> { } let (mut u, mut v) = (pool::take_u16(cw * ch), pool::take_u16(cw * ch)); for y in 0..ch {- let quads = row(src.chroma, src.stride, coy + y, cox * 4, cw * 4)?.as_chunks::<4>().0;+ let quads = row(src.chroma, src.chroma_stride, coy + y, cox * 4, cw * 4)?.as_chunks::<4>().0; u.extend(quads.iter().map(|q| u16::from_le_bytes([q[0], q[1]]) >> shift)); v.extend(quads.iter().map(|q| u16::from_le_bytes([q[2], q[3]]) >> shift)); }@@ -95,7 +102,7 @@ mod tests { let (stride, rows) = (10usize, 6usize); let luma: Vec<u8> = (0..stride * rows).map(|i| i as u8).collect(); let chroma: Vec<u8> = (0..stride * rows / 2).map(|i| 100 + i as u8).collect();- let f = to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride, rows }, &geometry((2, 2, 4, 4), 8)).unwrap();+ let f = to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride, chroma_stride: stride, rows }, &geometry((2, 2, 4, 4), 8)).unwrap(); let PixelData::Yuv8 { planes, chroma: c, .. } = &f.data else { panic!("8-bit") }; assert_eq!(*c, Chroma::C420); assert_eq!(&planes[0][..], &[22, 23, 24, 25, 32, 33, 34, 35, 42, 43, 44, 45, 52, 53, 54, 55]);@@ -104,6 +111,21 @@ mod tests { assert_eq!(&planes[2][..], &[113, 115, 123, 125]); } + #[test]+ fn chroma_rows_use_their_own_pitch() {+ // a 4x4 picture: luma 4 bytes per row, chroma padded to 6 (a derived VA-API image can+ // pitch its planes differently)+ let luma: Vec<u8> = (0..16).collect();+ let chroma: Vec<u8> = vec![100, 101, 102, 103, 0, 0, 110, 111, 112, 113, 0, 0];+ let f = to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride: 4, chroma_stride: 6, rows: 4 }, &geometry((0, 0, 4, 4), 8)).unwrap();+ let PixelData::Yuv8 { planes, .. } = &f.data else { panic!("8-bit") };+ assert_eq!(&planes[0][..], &luma[..]);+ assert_eq!(&planes[1][..], &[100, 102, 110, 112]);+ assert_eq!(&planes[2][..], &[101, 103, 111, 113]);+ // a chroma pitch too short for the picture is an error, not a short read+ assert!(to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride: 4, chroma_stride: 3, rows: 4 }, &geometry((0, 0, 4, 4), 8)).is_err());+ }+ #[test] fn p010_shifts_down_from_the_high_bits() { let (stride, rows) = (8usize, 2usize);@@ -115,25 +137,46 @@ mod tests { for (i, s) in chroma.as_chunks_mut::<2>().0.iter_mut().enumerate() { *s = ((500 + i as u16) << 6).to_le_bytes(); }- let f = to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride, rows }, &geometry((0, 0, 4, 2), 10)).unwrap();+ let f = to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride, chroma_stride: stride, rows }, &geometry((0, 0, 4, 2), 10)).unwrap(); let PixelData::Yuv16 { planes, bits, .. } = &f.data else { panic!("16-bit") }; assert_eq!(*bits, 10); assert_eq!(&planes[0][..], &[1, 2, 3, 4, 5, 6, 7, 8]); assert_eq!((&planes[1][..], &planes[2][..]), (&[500, 502][..], &[501, 503][..])); } + #[test]+ fn p010_chroma_rows_use_their_own_pitch() {+ // a 4x4 P010 picture: luma 8 bytes per row, chroma 8 bytes padded to 12+ let (stride, chroma_stride, rows) = (8usize, 12usize, 4usize);+ let mut luma = vec![0u8; stride * rows];+ for (i, s) in luma.as_chunks_mut::<2>().0.iter_mut().enumerate() {+ *s = (((i as u16) + 1) << 6).to_le_bytes();+ }+ let mut chroma = vec![0u8; chroma_stride * 2];+ for row in 0..2 {+ for i in 0..4 {+ let at = row * chroma_stride + i * 2;+ chroma[at..at + 2].copy_from_slice(&((500 + row as u16 * 10 + i as u16) << 6).to_le_bytes());+ }+ }+ let f = to_frame(&Biplanar { luma: &luma, chroma: &chroma, stride, chroma_stride, rows }, &geometry((0, 0, 4, 4), 10)).unwrap();+ let PixelData::Yuv16 { planes, .. } = &f.data else { panic!("16-bit") };+ assert_eq!(&planes[0][..], &(1..=16).collect::<Vec<u16>>()[..]);+ assert_eq!((&planes[1][..], &planes[2][..]), (&[500, 502, 510, 512][..], &[501, 503, 511, 513][..]));+ }+ #[test] fn surfaces_smaller_than_the_picture_are_errors() { let luma = vec![0u8; 64]; let chroma = vec![0u8; 32];- let src = Biplanar { luma: &luma, chroma: &chroma, stride: 8, rows: 8 };+ let src = Biplanar { luma: &luma, chroma: &chroma, stride: 8, chroma_stride: 8, rows: 8 }; for crop in [(0, 0, 9, 8), (0, 0, 8, 9), (4, 0, 8, 8), (0, 0, 0, 8), (u32::MAX, 0, 8, 8), (0, 0, 8, 8)] { let r = to_frame(&src, &geometry(crop, 8)); assert_eq!(r.is_ok(), crop == (0, 0, 8, 8), "{crop:?}"); } assert!(to_frame(&src, &geometry((0, 0, 4, 4), 12)).is_err()); // a short chroma plane is an error, not a panic- let short = Biplanar { luma: &luma, chroma: &chroma[..8], stride: 8, rows: 8 };+ let short = Biplanar { luma: &luma, chroma: &chroma[..8], stride: 8, chroma_stride: 8, rows: 8 }; assert!(to_frame(&short, &geometry((0, 0, 8, 8), 8)).is_err()); } }@@ -261,7 +261,7 @@ impl Readback { // SAFETY: see the `Map` call: `luma_len + chroma_len` bytes are mapped at `pData`. let all = unsafe { std::slice::from_raw_parts(mapped.pData as *const u8, luma_len + chroma_len) }; let (luma, chroma) = all.split_at(luma_len);- f(&Biplanar { luma, chroma, stride, rows })+ f(&Biplanar { luma, chroma, stride, chroma_stride: stride, rows }) })(); // SAFETY: the subresource was mapped above and nothing borrowed from it outlives `result`. unsafe { gpu.context.Unmap(staging, 0) };@@ -238,7 +238,7 @@ impl State { device.copy_to_host(address, &mut host)?; let luma = host.get(..luma_size).ok_or("NVDEC luma range is invalid")?; let chroma = host.get(luma_size..).ok_or("NVDEC chroma range is invalid")?;- to_frame(&Biplanar { luma, chroma, stride: pitch as usize, rows: self.coded.1 as usize }, &self.geometry)+ to_frame(&Biplanar { luma, chroma, stride: pitch as usize, chroma_stride: pitch as usize, rows: self.coded.1 as usize }, &self.geometry) } } fn mapped_layout(coded: (u32, u32), bits: u32, pitch: u32) -> Result<(usize, usize), String> {@@ -498,6 +498,7 @@ pub type vaEndPicture = unsafe extern "C" fn(dpy: VADisplay, context: VAContextI pub type vaSyncSurface = unsafe extern "C" fn(dpy: VADisplay, render_target: VASurfaceID) -> VAStatus; pub type vaCreateImage = unsafe extern "C" fn(dpy: VADisplay, format: *mut VAImageFormat, width: c_int, height: c_int, image: *mut VAImage) -> VAStatus; pub type vaDestroyImage = unsafe extern "C" fn(dpy: VADisplay, image: VAImageID) -> VAStatus;+pub type vaDeriveImage = unsafe extern "C" fn(dpy: VADisplay, surface: VASurfaceID, image: *mut VAImage) -> VAStatus; pub type vaGetImage = unsafe extern "C" fn(dpy: VADisplay, surface: VASurfaceID, x: c_int, y: c_int, width: c_uint, height: c_uint, image: VAImageID) -> VAStatus; pub type vaPutImage = unsafe extern "C" fn(@@ -30,7 +30,7 @@ struct Recorder { impl Recorder { fn new(count: usize) -> Self { let g = Geometry { crop: (0, 0, 2, 2), bits: 8, color: Default::default(), par: (1, 1) };- let frame = biplanar::to_frame(&Biplanar { luma: &[128; 4], chroma: &[128; 2], stride: 2, rows: 2 }, &g).unwrap();+ let frame = biplanar::to_frame(&Biplanar { luma: &[128; 4], chroma: &[128; 2], stride: 2, chroma_stride: 2, rows: 2 }, &g).unwrap(); Self { ids: (0..count as u32).map(|i| 0x100 + i).collect(), decoded: vec![false; count], pictures: 0, frame } } }@@ -46,6 +46,8 @@ struct Api { create_image: ffi::vaCreateImage, destroy_image: ffi::vaDestroyImage, get_image: ffi::vaGetImage,+ /// Maps a surface's own memory as an image (no copy into a second image); optional.+ derive_image: Option<ffi::vaDeriveImage>, put_image: ffi::vaPutImage, map_buffer: ffi::vaMapBuffer, unmap_buffer: ffi::vaUnmapBuffer,@@ -73,6 +75,14 @@ fn load() -> Result<Api, String> { *s.map_err(|e| format!("{}: {e}", stringify!($name)))? }}; }+ // A symbol an older libva may not have: absent is not an error (the caller declines).+ macro_rules! sym_opt {+ ($lib:expr, $name:ident) => {{+ // SAFETY: as `sym!`, but the pointer is kept only when the symbol resolves.+ let s = unsafe { $lib.get::<ffi::$name>(concat!(stringify!($name), "\0").as_bytes()) };+ s.ok().map(|s| *s)+ }};+ } Ok(Api { initialize: sym!(va, vaInitialize), terminate: sym!(va, vaTerminate),@@ -96,6 +106,7 @@ fn load() -> Result<Api, String> { create_image: sym!(va, vaCreateImage), destroy_image: sym!(va, vaDestroyImage), get_image: sym!(va, vaGetImage),+ derive_image: sym_opt!(va, vaDeriveImage), put_image: sym!(va, vaPutImage), map_buffer: sym!(va, vaMapBuffer), unmap_buffer: sym!(va, vaUnmapBuffer),@@ -254,6 +265,8 @@ pub struct Session { context: ffi::VAContextID, surfaces: Vec<VASurfaceID>, image: Option<VAImage>,+ /// Whether to read pictures through `vaDeriveImage` (cleared the first time it fails).+ derive: bool, /// Coded size of the surfaces (luma samples). size: (u32, u32), geometry: Geometry,@@ -321,7 +334,7 @@ impl Session { } let config = config.ok_or(why)?; // From here on `Drop` releases what was made.- let mut s = Self { display, config, context: ffi::VA_INVALID_ID, surfaces: Vec::new(), image: None, size, geometry };+ let mut s = Self { display, config, context: ffi::VA_INVALID_ID, surfaces: Vec::new(), image: None, derive: true, size, geometry }; let count_c = c_uint::try_from(count).map_err(|_| "too many surfaces")?; let mut surfaces = vec![ffi::VA_INVALID_SURFACE; count]; // SAFETY: `surfaces` has room for `count` ids; no attributes are passed (null, 0).@@ -431,6 +444,30 @@ impl Session { Ok(image) } + /// The picture in surface `index`, read through an image derived from the surface itself:+ /// the driver maps the decoded memory instead of copying it into a second image first, which+ /// is most of `vaGetImage`'s cost at 4K. An error leaves the caller to use `vaGetImage`.+ fn read_derived(&mut self, index: usize) -> Result<VideoFrame, String> {+ let derive = self.display.api.derive_image.ok_or("libva has no vaDeriveImage")?;+ let surface = *self.surfaces.get(index).ok_or("surface out of range")?;+ let api = self.display.api;+ // SAFETY: `surface` is one of this session's surfaces.+ let st = unsafe { (api.sync_surface)(self.display.dpy, surface) };+ self.display.check(st, "vaSyncSurface")?;+ let mut image = VAImage { image_id: ffi::VA_INVALID_ID, buf: ffi::VA_INVALID_ID, ..Default::default() };+ // SAFETY: `surface` was made on this display; `image` is a live output the driver fills in.+ let st = unsafe { derive(self.display.dpy, surface, &mut image) };+ self.display.check(st, "vaDeriveImage")?;+ let frame = self.read_image(&image);+ // SAFETY: the image was derived above on this display and is destroyed once; its buffer+ // has been unmapped by `read_image`.+ let st = unsafe { (api.destroy_image)(self.display.dpy, image.image_id) };+ // a derived image the driver cannot free would pile up: stop deriving (the caller falls+ // back to vaGetImage)+ self.display.check(st, "vaDestroyImage (derived image)")?;+ frame+ }+ /// Every sample of `image` set to mid-gray: 128 in NV12, 512 (0x8000 in the high bits of /// little-endian 16-bit words) in P010. fn write_gray(&self, image: &VAImage) -> Result<(), String> {@@ -497,12 +534,9 @@ fn biplanar_view<'a>(image: &VAImage, data: &'a [u8]) -> Result<Biplanar<'a>, St return Err(format!("the read-back image is not NV12 / P010 (fourcc {:#x}, {} planes)", image.format.fourcc, image.num_planes)); } let [p0, p1, _] = image.pitches;- if p0 != p1 {- return Err(format!("NV12 planes with different pitches ({p0}, {p1})"));- } let luma = data.get(image.offsets[0] as usize..).ok_or("NV12 luma offset out of range")?; let chroma = data.get(image.offsets[1] as usize..).ok_or("NV12 chroma offset out of range")?;- Ok(Biplanar { luma, chroma, stride: p0 as usize, rows: usize::from(image.height) })+ Ok(Biplanar { luma, chroma, stride: p0 as usize, chroma_stride: p1 as usize, rows: usize::from(image.height) }) } impl Accel for Session {@@ -531,6 +565,15 @@ impl Accel for Session { } fn read(&mut self, index: usize) -> Result<VideoFrame, String> {+ if self.derive {+ match self.read_derived(index) {+ Ok(frame) => return Ok(frame),+ Err(e) => {+ log::info!("VA-API: reading pictures with vaGetImage ({e})");+ self.derive = false;+ }+ }+ } let image = self.get_image(index)?; self.read_image(&image) }@@ -1,6 +1,6 @@ # Hardware parity -> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-10 · **Change:** major (new file: hardware decode / encode per platform and codec, GPU effects, devices; re-measured after NVDEC #549 and GPU export #421) · **Target:** Adobe Premiere Pro 2026 (26.5.2)+> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-24 · **Change:** minor (VA-API read-back through vaDeriveImage) · **Target:** Adobe Premiere Pro 2026 (26.5.2) Each hardware feature Premiere uses, per platform, against ours. Implementation detail lives in [`crates/platform/README.md`](../crates/platform/README.md) (decode / encode backends, the one@@ -27,7 +27,9 @@ release notes (Premiere does not run on Linux). Measured (`cargo xtask bench --hw off|auto`): M4 Pro, CPU per 2160p frame H.264 119 → 4.2 ms, HEVC 86 → 3.7 ms, 4K plays with no dropped frames at Full; Intel Iris Xe VA-API H.264 2160p-417 → 11.4 ms, HEVC Main 10 2160p 212 → 18.4 ms. Every decoded picture is read back to the CPU+417 → 11.4 ms, HEVC Main 10 2160p 212 → 18.4 ms. VA-API now reads pictures through `vaDeriveImage` where libva and the driver allow it (else `vaGetImage`): 4K HEVC+Main 10 decode 35 → 66–68 fps on an RX 9070 XT (measured during development, not with the bench;+[performance.md](performance.md#va-api-read-back-through-vaderiveimage)). Every decoded picture is read back to the CPU and re-uploaded to wgpu (**no zero-copy** yet), which Premiere avoids. ## Hardware encode@@ -85,3 +87,4 @@ one encoder ~4–8 h ([target-app-parity.md](target-app-parity.md#calibration)). | Date | Change | Summary | |---|---|---| | 2026-10-10 | major | Created from ROADMAP.md's performance-and-hardware row and `crates/platform/README.md`; added Linux NVDEC (#549), GPU export and Lumetri on the GPU (#421), Premiere's per-platform hardware list, devices and I/O |+| 2026-10-24 | minor | VA-API read-back through vaDeriveImage |@@ -495,10 +495,20 @@ software either way and show the run-to-run spread. The decoded pictures are ide What is left on the CPU per H.264 frame is the host side of stateless decoding (parsing, DPB, filling the VA buffers) and the read-back: `vaGetImage` into an NV12 image and the copy into-planar Y'CbCr. One picture is decoded at a time and read back as soon as the DPB outputs it, so+planar Y'CbCr (since then `vaDeriveImage`, below). One picture is decoded at a time and read back as soon as the DPB outputs it, so 4K throughput (56 fps) is bound by that round trip, not by the video engine; overlapping decode and read-back, or zero-copy into wgpu, would raise it. +### VA-API read-back through `vaDeriveImage`++The read-back maps the decoded surface's own NV12 / P010 memory (`vaDeriveImage`) instead of+copying it into a second image with `vaGetImage` first; a driver that cannot derive a surface+switches the session back to `vaGetImage` (logged once). Sequential decode of a 4K50 HEVC Main 10+file on an AMD Radeon RX 9070 XT (Mesa 26.2.5, radeonsi), before and after the change: 35 →+66–68 fps, pictures bit-identical to the software decoder. *Measured during development; the load+average was not recorded and it was not run through `cargo xtask bench`*, so treat it as an+indication until the bench table is re-measured.+ ### HEVC through VA-API (HW5 follow-up) Same machine and commands on 2026-10-09 with HEVC added, at a lower load (the software decoders ran