diff --git a/crates/platform/README.md b/crates/platform/README.md --- a/crates/platform/README.md +++ b/crates/platform/README.md @@ -143,7 +143,15 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox") - **What it takes:** H.264 in MP4 / MOV, 8-bit SDR, even picture sizes up to 8192, Baseline / Main / High (the level is chosen by the OS), constant or one-pass variable bitrate with the settings' target and ceiling, the keyframe distance (closed GOPs: every keyframe is an IDR picture). - Pictures are converted exactly like the built-in encoder's (BT.709, limited range), 4:2:0 NV12. +- **Pictures go in as the export's RGBA** (`32BGRA` buffers tagged BT.709 primaries and transfer) + and VideoToolbox converts them to limited-range BT.709 4:2:0 itself, so the export's encode + thread only copies the picture into the input buffer. The CPU conversion used to run on that + thread, where it competed with the render of the next frames for the thread pool and took longer + than the hardware encode: 20 ms a 1080p frame on an M2 Max, 88 % of the export's wall time, with + the hardware encoder idle meanwhile. A VideoToolbox that refuses RGB input (known from the first + picture; creating the session does not tell) gets that CPU conversion into NV12 instead, exactly + like the built-in encoder's (BT.709, limited range), and the stream starts over from its first + frame; `encode_rgba` returns the copy's time, which goes into `perf.stats` `export.stages.convertMs`. - **No B-frames.** On real 1080p footage the quality is the same with and without them (±0.3 dB at equal bitrate) and the bitrate lands closer to the target without them, so frame reordering is off: compressed frames come out in presentation order, with no composition offsets or edit list. diff --git a/crates/platform/src/hardware_encode.rs b/crates/platform/src/hardware_encode.rs --- a/crates/platform/src/hardware_encode.rs +++ b/crates/platform/src/hardware_encode.rs @@ -19,7 +19,7 @@ use filmcraft_export::{BitrateMode, EncodedPacket, EncoderFrame, ExportError, Ex use filmcraft_isobmff::{AvcConfig, HevcConfig, SampleEntry}; use filmcraft_time::FrameRate; -use crate::videotoolbox_encode::{ParamSets, VtCodec, VtConfig, VtEncoder, VtPacket, VtProfile, VtRate}; +use crate::videotoolbox_encode::{ParamSets, VtCodec, VtConfig, VtEncoder, VtInput, VtPacket, VtProfile, VtRate}; /// The hardware encoder's configuration for an export, or why it is not used for it. pub fn config_for(w: u32, h: u32, rate: FrameRate, s: &ExportSettings) -> Result { @@ -137,10 +137,10 @@ pub fn videotoolbox_encoder_factory( return None; } }; - match VtEncoder::new(config) { - Ok(vt) => { + match HardwareEncoder::open(config) { + Ok(enc) => { filmcraft_export::note_hw_encode_session(); - Some(Ok(Box::new(HardwareEncoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 }))) + Some(Ok(Box::new(enc))) } Err(why) => { log::info!("hardware {name} encoding declined: {why}"); @@ -150,6 +150,18 @@ pub fn videotoolbox_encoder_factory( } } +/// The session for `config`: one that takes the export's RGBA pictures as they are, so that +/// VideoToolbox converts them to 4:2:0 and the encode thread only copies (the CPU conversion ran +/// there before, where it competed with the render of the next frames for the thread pool and took +/// longer than the hardware encode itself); when that cannot be created, the planar session with +/// the CPU conversion. `rgba` stands for [`VtEncoder::with_rgba_input`] so tests can refuse it. +fn open_with(config: VtConfig, rgba: impl FnOnce(VtConfig) -> Result) -> Result { + rgba(config.clone()).or_else(|why| { + log::info!("VideoToolbox RGBA input declined ({why}): converting to 4:2:0 on the CPU"); + VtEncoder::new(config) + }) +} + /// The `hvcC` configuration of an HEVC stream: VideoToolbox's own record when it wrote one. fn hevc_config(p: &ParamSets) -> Result { let record = p.hvcc.as_deref().ok_or("VideoToolbox wrote no hvcC record for the stream")?; @@ -163,6 +175,7 @@ fn hevc_config(p: &ParamSets) -> Result { /// [`VtEncoder`] as a `VideoEncoder`: RGBA pictures in, MP4 samples out. pub struct HardwareEncoder { vt: VtEncoder, + /// The planar path's conversion buffers (empty while VideoToolbox converts the RGBA pictures). y: Vec, u: Vec, v: Vec, @@ -171,6 +184,35 @@ pub struct HardwareEncoder { } impl HardwareEncoder { + /// The encoder for `config` (see [`open_with`]), or why there is none. + pub fn open(config: VtConfig) -> Result { + Ok(Self::around(open_with(config, VtEncoder::with_rgba_input)?)) + } + + fn around(vt: VtEncoder) -> Self { + HardwareEncoder { vt, y: Vec::new(), u: Vec::new(), v: Vec::new(), decoded: 0 } + } + + /// Whether VideoToolbox converts the RGBA pictures itself (else the CPU converts them to 4:2:0 first). + pub fn takes_rgba(&self) -> bool { + self.vt.input() == VtInput::Bgra + } + + /// Replace the RGB session by a planar one for the rest of the export. Only for a session that + /// has produced nothing yet (the first picture), so the stream starts over from its first frame. + fn fall_back_to_planar(&mut self, why: &str) -> filmcraft_export::Result<()> { + log::info!("VideoToolbox did not take an RGBA picture ({why}): converting to 4:2:0 on the CPU"); + self.vt = VtEncoder::new(self.vt.config().clone()).map_err(ExportError::Encode)?; + Ok(()) + } + + /// Convert on the CPU and submit to the planar session. + fn encode_planar(&mut self, f: &EncoderFrame) -> filmcraft_export::Result> { + let (w, h) = (f.width as usize, f.height as usize); + filmcraft_export::timed(filmcraft_export::Stage::Convert, || filmcraft_export::rgba_to_yuv420_8(f.rgba, w, h, &mut self.y, &mut self.u, &mut self.v)); + self.vt.encode(f.index, &self.y, &self.u, &self.v).map_err(ExportError::Encode) + } + /// MP4 samples from compressed frames. Without frame reordering every frame is presented at /// its decode time, so composition offsets are 0; a frame whose time is not `n × duration` /// after the first (a dropped or reordered frame) is an error, never a file with wrong times. @@ -225,8 +267,25 @@ impl VideoEncoder for HardwareEncoder { if f.rgba.len() < w.saturating_mul(h).saturating_mul(4) { return Err(ExportError::Encode("the RGBA picture is smaller than its size".into())); } - filmcraft_export::timed(filmcraft_export::Stage::Convert, || filmcraft_export::rgba_to_yuv420_8(f.rgba, w, h, &mut self.y, &mut self.u, &mut self.v)); - let packets = self.vt.encode(f.index, &self.y, &self.u, &self.v).map_err(ExportError::Encode)?; + let packets = if self.takes_rgba() { + match self.vt.encode_rgba(f.index, f.rgba) { + Ok((packets, copied)) => { + // the CPU's share of the conversion: the copy into the input buffer + filmcraft_export::note_stage(filmcraft_export::Stage::Convert, copied); + packets + } + // Whether this VideoToolbox converts RGB itself is only known from the first picture + // (the session is created without complaint either way). Nothing has come out of it + // yet, so the export goes on from this frame with the planar path, the CPU converting. + Err(why) if f.index == 0 => { + self.fall_back_to_planar(&why)?; + self.encode_planar(f)? + } + Err(why) => return Err(ExportError::Encode(why)), + } + } else { + self.encode_planar(f)? + }; filmcraft_export::note_hw_encode_frame(); // the muxer needs the parameter sets after the first group of pictures if f.index == 0 { @@ -243,3 +302,111 @@ impl VideoEncoder for HardwareEncoder { self.samples(packets) } } + +#[cfg(test)] +mod tests { + use super::*; + + const W: u32 = 320; + const H: u32 = 180; + + fn config() -> VtConfig { + VtConfig { + width: W, + height: H, + timescale: 30, + frame_duration: 1, + keyframe_interval: 30, + profile: VtProfile::High, + rate: VtRate::Vbr { target_kbps: 2000, max_kbps: 3000 }, + } + } + + /// A gradient picture with a bar at `i`. + fn picture(i: u64) -> Vec { + let (w, h) = (W as usize, H as usize); + let mut px = vec![255u8; w * h * 4]; + for y in 0..h { + for x in 0..w { + let o = (y * w + x) * 4; + px[o] = (x * 255 / w) as u8; + px[o + 1] = (y * 255 / h) as u8; + px[o + 2] = if x / 10 == i as usize { 255 } else { 40 }; + } + } + px + } + + /// Three pictures through the `VideoEncoder` adapter: one compressed frame each. + fn encodes_three(enc: &mut HardwareEncoder) { + let mut packets = Vec::new(); + for i in 0..3u64 { + let px = picture(i); + packets.extend(enc.encode(&EncoderFrame { width: W, height: H, rgba: &px, hdr: None, index: i }).expect("encode")); + } + packets.extend(enc.flush().expect("flush")); + assert_eq!(packets.len(), 3); + assert!(packets[0].key); + } + + #[test] + fn rgba_input_is_tried_first() { + let mut attempted = false; + let opened = open_with(config(), |c| { + attempted = true; + assert_eq!((c.width, c.height), (W, H)); + VtEncoder::with_rgba_input(c) + }); + assert!(attempted); + match opened { + Ok(vt) => assert_eq!(vt.input(), VtInput::Bgra), + Err(why) => eprintln!("SKIPPED: no VideoToolbox hardware encoder ({why})"), + } + } + + /// Where the RGB session cannot be created, the planar one takes the export (the CPU converts). + #[test] + fn refused_rgba_input_falls_back_to_the_planar_session() { + let opened = open_with(config(), |_| Err("no RGB input (test)".into())); + let Ok(vt) = opened else { + eprintln!("SKIPPED: no VideoToolbox hardware encoder"); + return; + }; + assert_eq!(vt.input(), VtInput::Nv12); + let mut enc = HardwareEncoder::around(vt); + assert!(!enc.takes_rgba()); + encodes_three(&mut enc); + } + + /// Where the RGB session refuses the first picture, the encoder starts over with a planar session + /// and the export goes on from its first frame. + #[test] + fn a_refused_first_picture_restarts_with_the_planar_session() { + let Ok(vt) = VtEncoder::with_rgba_input(config()) else { + eprintln!("SKIPPED: no VideoToolbox hardware encoder"); + return; + }; + let mut enc = HardwareEncoder::around(vt); + assert!(enc.takes_rgba()); + enc.fall_back_to_planar("test").expect("the planar session opens"); + assert!(!enc.takes_rgba()); + encodes_three(&mut enc); + } + + /// A picture the RGB session fails after the first frame is an error, not a silent restart (the + /// stream already has frames the planar session could not continue). + #[test] + fn a_later_failure_is_an_error() { + let Ok(vt) = VtEncoder::with_rgba_input(config()) else { + eprintln!("SKIPPED: no VideoToolbox hardware encoder"); + return; + }; + let mut enc = HardwareEncoder::around(vt); + let px = picture(0); + enc.encode(&EncoderFrame { width: W, height: H, rgba: &px, hdr: None, index: 0 }).expect("first frame"); + // a short picture is refused before it reaches VideoToolbox; so is one of another size + assert!(enc.encode(&EncoderFrame { width: W, height: H, rgba: &px[..100], hdr: None, index: 1 }).is_err()); + assert!(enc.encode(&EncoderFrame { width: W + 2, height: H, rgba: &px, hdr: None, index: 1 }).is_err()); + assert!(enc.takes_rgba(), "no restart after the first frame"); + } +} diff --git a/crates/platform/src/videotoolbox_encode.rs b/crates/platform/src/videotoolbox_encode.rs --- a/crates/platform/src/videotoolbox_encode.rs +++ b/crates/platform/src/videotoolbox_encode.rs @@ -9,11 +9,17 @@ //! is declined up front. Frame reordering (B-frames) is off, so compressed frames come out in //! presentation order with decode time = presentation time. //! -//! Pictures go in as 8-bit 4:2:0 (NV12) pixel buffers filled from planar BT.709 limited-range -//! Y'CbCr, the layout of `filmcraft_export::rgba_to_yuv420_8`, so the colours match the built-in -//! encoder's. VideoToolbox encodes asynchronously; its output callback (on a VideoToolbox thread) -//! copies each compressed frame (length-prefixed NAL units) into a queue that [`VtEncoder::encode`] -//! and [`VtEncoder::flush`] drain. +//! Pictures go in one of two ways ([`VtInput`]). On the RGB path ([`VtEncoder::with_rgba_input`], +//! [`VtEncoder::encode_rgba`]) the export's straight sRGB RGBA8 picture is copied into a `32BGRA` +//! pixel buffer tagged BT.709, and VideoToolbox converts it to the encoder's 4:2:0 itself, off the +//! export's encode thread: there the CPU conversion competed with the render of the next frames for +//! the thread pool and took longer than the hardware encode (20 ms a 1080p frame), leaving the +//! encoder waiting. On the planar path ([`VtEncoder::new`], [`VtEncoder::encode`]) pictures go in as +//! 8-bit 4:2:0 (NV12) pixel buffers filled from planar BT.709 limited-range Y'CbCr, the layout of +//! `filmcraft_export::rgba_to_yuv420_8`, so the colours match the built-in encoder's; it is the +//! fallback for a VideoToolbox that does not take RGB input. VideoToolbox encodes asynchronously; +//! its output callback (on a VideoToolbox thread) copies each compressed frame (length-prefixed NAL +//! units) into a queue that [`VtEncoder::encode_picture`] and [`VtEncoder::flush`] drain. //! //! The first frame is completed straight away: the parameter sets the container needs (SPS / PPS //! for H.264, VPS / SPS / PPS and VideoToolbox's own `hvcC` record for HEVC) come with the first @@ -31,10 +37,11 @@ use objc2_core_media::{ kCMVideoCodecType_HEVC, }; use objc2_core_video::{ - CVImageBuffer, CVPixelBufferCreate, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeightOfPlane, - CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress, CVPixelBufferLockFlags, CVPixelBufferPool, CVPixelBufferUnlockBaseAddress, - kCVImageBufferColorPrimaries_ITU_R_709_2, kCVImageBufferTransferFunction_ITU_R_709_2, kCVImageBufferYCbCrMatrix_ITU_R_709_2, kCVPixelBufferHeightKey, - kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, + CVAttachmentMode, CVImageBuffer, CVPixelBufferCreate, CVPixelBufferGetBaseAddress, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRow, + CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeight, CVPixelBufferGetHeightOfPlane, CVPixelBufferGetWidth, CVPixelBufferGetWidthOfPlane, + CVPixelBufferLockBaseAddress, CVPixelBufferLockFlags, CVPixelBufferPool, CVPixelBufferUnlockBaseAddress, kCVImageBufferColorPrimaries_ITU_R_709_2, + kCVImageBufferColorPrimariesKey, kCVImageBufferTransferFunction_ITU_R_709_2, kCVImageBufferTransferFunctionKey, kCVImageBufferYCbCrMatrix_ITU_R_709_2, + kCVPixelBufferHeightKey, kCVPixelBufferIOSurfacePropertiesKey, kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, }; use objc2_video_toolbox::{ VTCompressionSession, VTEncodeInfoFlags, VTSessionCopyProperty, VTSessionSetProperty, kVTCompressionPropertyKey_AllowFrameReordering, @@ -46,8 +53,10 @@ use objc2_video_toolbox::{ kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder, }; -/// The pixel format of the buffers we hand over: biplanar Y + interleaved CbCr, video range. +/// The pixel format of the planar input path: biplanar Y + interleaved CbCr, video range. const NV12_VIDEO: u32 = u32::from_be_bytes(*b"420v"); +/// The pixel format of the RGB input path: 8-bit B, G, R, A bytes (`kCVPixelFormatType_32BGRA`). +const BGRA_8: u32 = u32::from_be_bytes(*b"BGRA"); /// Largest compressed frame accepted from the callback (a sanity cap on a size we do not control). const MAX_FRAME_BYTES: usize = 256 << 20; /// Largest picture the hardware encoder is asked for. @@ -88,6 +97,16 @@ pub enum VtRate { Vbr { target_kbps: u32, max_kbps: u32 }, } +/// The pixel format a session takes its pictures in. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum VtInput { + /// Biplanar 8-bit 4:2:0 (NV12), filled from planar Y'CbCr by [`VtEncoder::encode`]. + Nv12, + /// 8-bit BGRA, filled from the export's straight RGBA8 picture by [`VtEncoder::encode_rgba`]; + /// VideoToolbox converts it to 4:2:0. + Bgra, +} + /// What to encode and how. Times are in a timescale of `timescale` units per second, one frame /// lasting `frame_duration` units (the frame rate's numerator and denominator). #[derive(Clone, Debug)] @@ -400,11 +419,42 @@ fn fill(pb: &CVImageBuffer, w: usize, h: usize, y: &[u8], u: &[u8], v: &[u8]) -> Ok(()) } +/// Copy a straight RGBA8 picture (`w`×`h`, tightly packed) into a BGRA buffer, swapping R and B. +fn fill_bgra(pb: &CVImageBuffer, w: usize, h: usize, rgba: &[u8]) -> Result<(), String> { + let row_bytes = w.checked_mul(4).ok_or("picture width overflows")?; + if rgba.len() < row_bytes.checked_mul(h).ok_or("picture size overflows")? { + return Err("the RGBA picture is smaller than its size".into()); + } + // SAFETY: `pb` is a valid pixel buffer (the caller holds it retained). + if unsafe { CVPixelBufferLockBaseAddress(pb, CVPixelBufferLockFlags(0)) } != 0 { + return Err("cannot lock the input buffer".into()); + } + let lock = Locked(pb); + let base = CVPixelBufferGetBaseAddress(pb) as *mut u8; + let stride = CVPixelBufferGetBytesPerRow(pb); + if base.is_null() || stride < row_bytes || CVPixelBufferGetWidth(pb) < w || CVPixelBufferGetHeight(pb) < h { + return Err("the input buffer is smaller than the picture".into()); + } + let len = stride.checked_mul(h).ok_or("buffer size overflows")?; + // SAFETY: the buffer's base address is locked for writing while `lock` lives and the slice is + // only used before it is dropped; CoreVideo maps `bytes_per_row * height` bytes from the base, + // of which `len` covers the `h` rows written. + let dst = unsafe { std::slice::from_raw_parts_mut(base, len) }; + for (row, src) in dst.chunks_mut(stride).zip(rgba.chunks_exact(row_bytes)).take(h) { + for (d, s) in row[..row_bytes].as_chunks_mut::<4>().0.iter_mut().zip(src.as_chunks::<4>().0) { + *d = [s[2], s[1], s[0], s[3]]; + } + } + drop(lock); + Ok(()) +} + /// A hardware H.264 or HEVC encoder: feed pictures, collect compressed frames (in presentation /// order, which is decoding order since nothing is reordered). pub struct VtEncoder { session: Session, config: VtConfig, + input: VtInput, pool: Option>, attrs: CFRetained>, /// Pictures submitted so far. @@ -454,8 +504,21 @@ fn require(status: i32, what: &str) -> Result<(), String> { } impl VtEncoder { - /// Create the session, or say why hardware encoding is not used for this configuration. + /// Create a session that takes planar Y'CbCr pictures ([`VtEncoder::encode`]), or say why + /// hardware encoding is not used for this configuration. pub fn new(config: VtConfig) -> Result { + Self::with_input(config, VtInput::Nv12) + } + + /// Create a session that takes the export's RGBA pictures as they are ([`VtEncoder::encode_rgba`]): + /// VideoToolbox converts them to 4:2:0 itself, so the export's encode thread only copies. Whether + /// this VideoToolbox converts RGB is known from the first picture, not from creating the session: + /// `hardware_encode` replaces a session that refuses it with a planar one ([`VtEncoder::new`]). + pub fn with_rgba_input(config: VtConfig) -> Result { + Self::with_input(config, VtInput::Bgra) + } + + fn with_input(config: VtConfig, input: VtInput) -> Result { let (w, h) = (config.width, config.height); if w == 0 || h == 0 || w > MAX_SIDE || h > MAX_SIDE { return Err(format!("picture size {w}x{h}")); @@ -474,7 +537,11 @@ impl VtEncoder { let require_key: &CFString = unsafe { kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder }; let spec = CFDictionary::::from_slices(&[require_key], &[CFBoolean::new(true).as_ref()]); - let (pf, cw, ch) = (CFNumber::new_i32(NV12_VIDEO as i32), CFNumber::new_i32(w as i32), CFNumber::new_i32(h as i32)); + let pixel_format = match input { + VtInput::Nv12 => NV12_VIDEO, + VtInput::Bgra => BGRA_8, + }; + let (pf, cw, ch) = (CFNumber::new_i32(pixel_format as i32), CFNumber::new_i32(w as i32), CFNumber::new_i32(h as i32)); let io = CFDictionary::::from_slices(&[], &[]); // SAFETY: reading immutable framework constants. let (pf_key, w_key, h_key, io_key): (&CFString, &CFString, &CFString, &CFString) = @@ -511,7 +578,7 @@ impl VtEncoder { require(unsafe { session.session.prepare_to_encode_frames() }, "to prepare the encoder")?; // SAFETY: the session is valid; the pool (if any) is returned retained. let pool = unsafe { session.session.pixel_buffer_pool() }; - Ok(VtEncoder { session, config, pool, attrs, submitted: 0 }) + Ok(VtEncoder { session, config, input, pool, attrs, submitted: 0 }) } fn configure(session: &VTCompressionSession, c: &VtConfig) -> Result<(), String> { @@ -614,6 +681,7 @@ impl VtEncoder { self.session.shared.params.lock().unwrap_or_else(PoisonError::into_inner).clone() } + /// An input buffer in the session's pixel format (from its pool when it has one). fn new_buffer(&self) -> Result, String> { let mut raw: *mut CVImageBuffer = std::ptr::null_mut(); let status = match &self.pool { @@ -625,7 +693,10 @@ impl VtEncoder { None, self.config.width as usize, self.config.height as usize, - NV12_VIDEO, + match self.input { + VtInput::Nv12 => NV12_VIDEO, + VtInput::Bgra => BGRA_8, + }, Some(self.attrs.as_ref()), NonNull::from(&mut raw), ) @@ -636,16 +707,68 @@ impl VtEncoder { Ok(unsafe { CFRetained::from_raw(raw) }) } - /// Submit picture number `index` (planar BT.709 limited-range Y'CbCr, tightly packed) and return - /// the compressed frames that came out meanwhile. + /// Submit picture number `index` (planar BT.709 limited-range Y'CbCr, tightly packed) to a + /// planar session ([`VtEncoder::new`]) and return the compressed frames that came out meanwhile. pub fn encode(&mut self, index: u64, y: &[u8], u: &[u8], v: &[u8]) -> Result, String> { + if self.input != VtInput::Nv12 { + return Err("this session takes RGBA pictures, not Y'CbCr planes".into()); + } let buffer = self.new_buffer()?; fill(&buffer, self.config.width as usize, self.config.height as usize, y, u, v)?; + self.encode_picture(index, &buffer) + } + + /// Submit picture number `index` (the export's straight RGBA8 picture, `width × height × 4` + /// bytes, tightly packed) to an RGB session ([`VtEncoder::with_rgba_input`]) and return the + /// compressed frames that came out meanwhile, with the time the copy into the input buffer + /// took (the CPU's whole share of the conversion; VideoToolbox does the rest). The buffer is + /// tagged BT.709 primaries and transfer, like the stream, so VideoToolbox applies no transfer + /// conversion: only the session's BT.709 matrix to limited-range 4:2:0, as the planar path does + /// on the CPU. + pub fn encode_rgba(&mut self, index: u64, rgba: &[u8]) -> Result<(Vec, std::time::Duration), String> { + if self.input != VtInput::Bgra { + return Err("this session takes Y'CbCr planes, not RGBA pictures".into()); + } + let t = std::time::Instant::now(); + let buffer = self.new_buffer()?; + fill_bgra(&buffer, self.config.width as usize, self.config.height as usize, rgba)?; + // SAFETY: reading immutable framework constants. + let (primaries_key, transfer_key, primaries, transfer): (&CFString, &CFString, &CFString, &CFString) = unsafe { + ( + kCVImageBufferColorPrimariesKey, + kCVImageBufferTransferFunctionKey, + kCVImageBufferColorPrimaries_ITU_R_709_2, + kCVImageBufferTransferFunction_ITU_R_709_2, + ) + }; + // SAFETY: `buffer` is a valid pixel buffer we hold retained; the keys are framework + // constants and the values the CFStrings CoreVideo documents for them. + unsafe { + buffer.set_attachment(primaries_key, primaries.as_ref(), CVAttachmentMode::ShouldPropagate); + buffer.set_attachment(transfer_key, transfer.as_ref(), CVAttachmentMode::ShouldPropagate); + } + let copied = t.elapsed(); + Ok((self.encode_picture(index, &buffer)?, copied)) + } + + /// The format pictures go in as. + pub fn input(&self) -> VtInput { + self.input + } + + /// Pictures submitted so far. + pub fn submitted(&self) -> u64 { + self.submitted + } + + /// Submit a filled input buffer as picture number `index` and return the compressed frames + /// that came out meanwhile. + fn encode_picture(&mut self, index: u64, buffer: &CVImageBuffer) -> Result, String> { let value = i64::try_from(index).ok().and_then(|i| i.checked_mul(i64::from(self.config.frame_duration))).ok_or("frame index out of range")?; let (pts, duration) = (cm_time(value, self.config.timescale), cm_time(i64::from(self.config.frame_duration), self.config.timescale)); // SAFETY: the session and buffer are valid (VideoToolbox retains the buffer as long as it // needs it); no frame properties, refcon or info-flags out-pointer. - let status = unsafe { self.session.session.encode_frame(&buffer, pts, duration, None, std::ptr::null_mut(), std::ptr::null_mut()) }; + let status = unsafe { self.session.session.encode_frame(buffer, pts, duration, None, std::ptr::null_mut(), std::ptr::null_mut()) }; if status != 0 { return Err(format!("VTCompressionSessionEncodeFrame failed ({status})")); } diff --git a/crates/platform/tests/it/hardware_encode.rs b/crates/platform/tests/it/hardware_encode.rs --- a/crates/platform/tests/it/hardware_encode.rs +++ b/crates/platform/tests/it/hardware_encode.rs @@ -16,8 +16,8 @@ use filmcraft_frame::PixelData; use filmcraft_isobmff::{CodecConfig, SampleEntry}; use filmcraft_media::generators::GeneratorSource; use filmcraft_media::{DemoScene, MediaSource}; -use filmcraft_platform::hardware_encode::{config_for, hevc_available, videotoolbox_encoder_factory}; -use filmcraft_platform::videotoolbox_encode::{VtConfig, VtEncoder, VtProfile, VtRate}; +use filmcraft_platform::hardware_encode::{HardwareEncoder, config_for, hevc_available, videotoolbox_encoder_factory}; +use filmcraft_platform::videotoolbox_encode::{VtConfig, VtEncoder, VtInput, VtProfile, VtRate}; use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind}; use filmcraft_render::SourceMap; use filmcraft_time::{FrameRate, Tick, TimeRange}; @@ -100,6 +100,197 @@ fn psnr(a: &[u8], b: &[u8]) -> f64 { if mse == 0.0 { 99.0 } else { 10.0 * (255.0f64 * 255.0 / mse).log10() } } +fn mean_abs_diff(a: &[u8], b: &[u8]) -> f64 { + assert_eq!(a.len(), b.len()); + a.iter().zip(b).map(|(x, y)| (f64::from(*x) - f64::from(*y)).abs()).sum::() / a.len() as f64 +} + +/// Decode `packets` (decoding order, no reordering) with our software decoder: every picture's +/// Y, Cb and Cr planes in presentation order. +fn decode_planes(entry: &SampleEntry, packets: &[EncodedPacket]) -> Vec<[Vec; 3]> { + let mut dec = filmcraft_codecs::software_video_decoder(entry).expect("software decoder"); + let mut out = Vec::new(); + let take = |frames: Vec, out: &mut Vec<(i64, [Vec; 3])>| { + for f in frames { + let PixelData::Yuv8 { planes, .. } = &f.frame.data else { panic!("expected 8-bit planes") }; + out.push((f.pts, [planes[0].to_vec(), planes[1].to_vec(), planes[2].to_vec()])); + } + }; + for (i, p) in packets.iter().enumerate() { + take(dec.decode(&p.data, i as i64).expect("decode"), &mut out); + } + take(dec.flush(), &mut out); + out.sort_by_key(|(pts, _)| *pts); + out.into_iter().map(|(_, planes)| planes).collect() +} + +/// The sample entry of a session's stream (known once its first frame came out). +fn entry_of(vt: &VtEncoder) -> SampleEntry { + let p = vt.parameter_sets().expect("parameter sets after the first frame"); + SampleEntry::avc(filmcraft_isobmff::AvcConfig::new(p.sps, p.pps, 4), W as u16, H as u16) +} + +/// RGBA pictures handed to VideoToolbox as they are (`32BGRA`, VideoToolbox converting) encode the +/// same pictures as the planar path (our own BT.709 limited-range conversion into NV12): every +/// plane of both decodes is within a few codes of our conversion of the source. A swapped channel, +/// a full-range conversion, another matrix or a transfer-function conversion would be tens of codes +/// off in some plane. The hardware output differs between the two paths by rate control and chroma +/// filtering only, so the quality is compared, not the bytes. +#[test] +fn rgba_pictures_encode_like_the_cpu_conversion() { + let cfg = config_for(W, H, FPS, &hardware_settings()).expect("a hardware configuration"); + let Ok(mut rgb) = VtEncoder::with_rgba_input(cfg.clone()) else { + eprintln!("SKIPPED: no VideoToolbox hardware H.264 encoder"); + return; + }; + let mut planar = VtEncoder::new(cfg).expect("the planar session opens where the RGB one does"); + assert_eq!((rgb.input(), planar.input()), (VtInput::Bgra, VtInput::Nv12)); + let n = 24u64; + let (mut a, mut b) = (Vec::new(), Vec::new()); + let (mut y, mut u, mut v) = (Vec::new(), Vec::new(), Vec::new()); + let mut copy = std::time::Duration::ZERO; + for i in 0..n { + let px = picture(i); + match rgb.encode_rgba(i, &px) { + Ok((packets, copied)) => { + a.extend(packets); + copy += copied; + } + Err(why) => { + eprintln!("SKIPPED: this VideoToolbox does not take RGBA pictures ({why})"); + return; + } + } + rgba_to_yuv420_8(&px, W as usize, H as usize, &mut y, &mut u, &mut v); + b.extend(planar.encode(i, &y, &u, &v).expect("planar encode")); + } + a.extend(rgb.flush().expect("flush")); + b.extend(planar.flush().expect("flush")); + let a: Vec = a.into_iter().map(|p| EncodedPacket { data: p.data, key: p.key, duration: 1, composition_offset: 0 }).collect(); + let b: Vec = b.into_iter().map(|p| EncodedPacket { data: p.data, key: p.key, duration: 1, composition_offset: 0 }).collect(); + assert_eq!((a.len() as u64, b.len() as u64), (n, n)); + let (da, db) = (decode_planes(&entry_of(&rgb), &a), decode_planes(&entry_of(&planar), &b)); + assert_eq!((da.len() as u64, db.len() as u64), (n, n)); + let mut worst = [99.0f64; 3]; + let mut gap = [0.0f64; 3]; + let mut apart = [0.0f64; 3]; + for i in 0..n as usize { + rgba_to_yuv420_8(&picture(i as u64), W as usize, H as usize, &mut y, &mut u, &mut v); + for (c, reference) in [&y, &u, &v].into_iter().enumerate() { + let (pa, pb) = (psnr(&da[i][c], reference), psnr(&db[i][c], reference)); + worst[c] = worst[c].min(pa); + gap[c] = gap[c].max(pb - pa); + apart[c] = apart[c].max(mean_abs_diff(&da[i][c], &db[i][c])); + } + } + // The planar path's chroma is our conversion itself (2×2 average), so against that reference it + // scores near-perfectly where VideoToolbox's filtered chroma siting scores ~40 dB: the gap is + // reported, the bounds are the absolute ones. (M2 Max: Y/Cb/Cr 52 / 41 / 37 dB, paths 0.2 / 0.6 / + // 0.8 codes apart; a swapped channel or a range or matrix mistake is tens of codes.) + eprintln!( + "RGBA input: worst PSNR Y/Cb/Cr {worst:.1?} dB, at most {gap:.1?} dB under the planar path, mean |Δ| between the paths {apart:.2?} codes; copy {:.2} ms a frame", + copy.as_secs_f64() * 1000.0 / n as f64 + ); + for c in 0..3 { + assert!(worst[c] > 30.0, "plane {c}: worst PSNR {:.1} dB against our conversion", worst[c]); + assert!(apart[c] < 2.0, "plane {c}: the paths are {:.2} codes apart on average", apart[c]); + } + assert!(gap[0] < 3.0, "luma {:.1} dB below the planar path", gap[0]); +} + +/// The adapter the factory registers prefers RGB input; the pictures it writes decode at the same quality. +#[test] +fn the_encoder_takes_rgba_pictures_where_videotoolbox_converts() { + let cfg = config_for(W, H, FPS, &hardware_settings()).expect("a hardware configuration"); + let Ok(enc) = HardwareEncoder::open(cfg) else { + eprintln!("SKIPPED: no VideoToolbox hardware H.264 encoder"); + return; + }; + assert!(enc.takes_rgba(), "the RGB session is tried first"); + let mut enc: Box = Box::new(enc); + let n = 20u64; + let packets = encode_all(&mut enc, n); + assert_eq!(packets.len() as u64, n); + let decoded = decode(&enc.sample_entry(), &packets); + let (mut y, mut u, mut v) = (Vec::new(), Vec::new(), Vec::new()); + let mut worst = 99.0f64; + for (i, (_, luma)) in decoded.iter().enumerate() { + rgba_to_yuv420_8(&picture(i as u64), W as usize, H as usize, &mut y, &mut u, &mut v); + worst = worst.min(psnr(&y, luma)); + } + assert!(worst > 30.0, "worst luma PSNR {worst:.1} dB"); +} + +/// Hostile RGBA pictures and the wrong kind of picture for a session give errors, never crashes. +#[test] +fn rgba_input_refuses_what_does_not_fit() { + let cfg = config_for(W, H, FPS, &hardware_settings()).expect("a hardware configuration"); + let Ok(mut rgb) = VtEncoder::with_rgba_input(cfg.clone()) else { + eprintln!("SKIPPED: no VideoToolbox hardware H.264 encoder"); + return; + }; + let (w, h) = (W as usize, H as usize); + assert!(calm("empty", || rgb.encode_rgba(0, &[])).is_err()); + assert!(calm("short", || rgb.encode_rgba(0, &vec![0; w * h * 4 - 1])).is_err()); + assert!(calm("planes to an RGB session", || rgb.encode(0, &vec![16; w * h], &vec![128; w * h / 4], &vec![128; w * h / 4])).is_err()); + let mut planar = VtEncoder::new(cfg).expect("planar session"); + assert!(calm("RGBA to a planar session", || planar.encode_rgba(0, &vec![0; w * h * 4])).is_err()); + // a picture larger than its size is fine (the extra bytes are ignored) + let _ = calm("long", || rgb.encode_rgba(0, &vec![0; w * h * 4 + 64])); + let _ = calm("flush", || rgb.flush()); +} + +/// An export at another frame rate than the sequence's (frames are repeated or dropped on the way) and +/// a cancelled export followed by a complete one in the same process: the hardware path must give the +/// asked-for frame count, release its sessions and start clean again. +#[test] +fn rate_conversion_and_cancel_then_export_again() { + filmcraft_platform::register(); + let probe = ExportSettings { include_audio: false, ..hardware_settings() }; + if hardware(&probe, W, H).is_none() { + return; + } + let (frames, (p, seq, m)) = (60i64, project(60)); + // 60 frames at 30 fps become 120 at 60 fps and 50 at 25 fps + for (rate, want) in [(FrameRate::FPS_60, 120usize), (FrameRate::FPS_25, 50)] { + let path = tmp(&format!("rate-{}.mp4", rate.num)); + let s = ExportSettings { path: path.clone(), frame_rate: Some(rate), ..probe.clone() }; + let report = export(&p, seq, &s, &m, &Progress::default()).expect("export"); + assert_eq!(report.frames as usize, want, "{rate:?}"); + let (samples, pictures, _) = inspect(&path); + assert_eq!((samples, pictures), (want, want), "{rate:?}"); + } + // cancel from another thread once a few frames are done + let before = hw_encode_stats(); + let path = tmp("cancelled.mp4"); + let s = ExportSettings { path: path.clone(), ..probe.clone() }; + let progress = Progress::default(); + let result = std::thread::scope(|scope| { + scope.spawn(|| { + while progress.done.load(std::sync::atomic::Ordering::Relaxed) < 8 && !progress.finished.load(std::sync::atomic::Ordering::Relaxed) { + std::thread::sleep(std::time::Duration::from_millis(2)); + } + progress.cancel.store(true, std::sync::atomic::Ordering::Relaxed); + }); + export(&p, seq, &s, &m, &progress) + }); + match result { + Err(filmcraft_export::ExportError::Cancelled) => {} + Ok(r) => eprintln!("the export finished before the cancel landed ({} frames)", r.frames), + Err(e) => panic!("a cancelled export must report Cancelled or finish, not {e}"), + } + // and the next export in the same process is complete + let path = tmp("after-cancel.mp4"); + let s = ExportSettings { path: path.clone(), ..probe }; + let report = export(&p, seq, &s, &m, &Progress::default()).expect("export after a cancel"); + assert_eq!(report.frames, frames as u64); + let (samples, pictures, _) = inspect(&path); + assert_eq!((samples, pictures), (frames as usize, frames as usize)); + let after = hw_encode_stats(); + assert!(after.sessions >= before.sessions + 2, "both exports opened a hardware session"); + assert_eq!(after.declined, before.declined, "nothing was declined to the software encoder"); +} + #[test] fn declines_what_the_hardware_path_does_not_take() { let ok = hardware_settings(); diff --git a/docs/architecture.md b/docs/architecture.md --- a/docs/architecture.md +++ b/docs/architecture.md @@ -298,6 +298,12 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see `filmcraft_export::register_format_probe`, so the H.265 format exists where an HEVC encoder does, and choosing it is the opt-in (the Hardware encoding toggle governs H.264 only). With no software H.265 encoder, what NVENC declines is counted and ends the export with an error that says why. + On macOS, `platform::hardware_encode` (VideoToolbox H.264, opt-in the same way, and H.265) is + registered likewise. It too takes the RGBA pictures as they are (`32BGRA` buffers tagged BT.709) + and VideoToolbox converts them to 4:2:0, so the encode thread only copies: converting on the CPU + there competed with the render of the next frames for the thread pool and cost more than the + hardware encode (20 ms a 1080p frame). A VideoToolbox that refuses RGB input on the first picture + gets the CPU's conversion into NV12 instead, and the export goes on. - **Compositor.** `render` is the reference for monitors, thumbnails and export. `render::plan` turns a frame into GPU layers, each with its opacity and blend mode. All 27 blend modes run on the GPU (`filmcraft-gpu`): Normal and Dissolve with fixed-function "over" blending, the others by diff --git a/docs/hardware-parity.md b/docs/hardware-parity.md --- a/docs/hardware-parity.md +++ b/docs/hardware-parity.md @@ -34,7 +34,7 @@ and re-uploaded to wgpu (**no zero-copy** yet), which Premiere avoids. | Codec | Premiere macOS | **Ours macOS** | Premiere Windows | **Ours Windows** | **Ours Linux** | |---|---|---|---|---|---| -| H.264 8-bit | yes | **yes** (VideoToolbox, opt-in, no B-frames) | Intel QSV, NVENC, AMD | **NVENC only** (opt-in) | **NVENC only** | +| H.264 8-bit | yes | **yes** (VideoToolbox, opt-in, no B-frames; takes the RGBA pictures and converts them itself) | Intel QSV, NVENC, AMD | **NVENC only** (opt-in) | **NVENC only** | | H.264 10-bit / 4:2:2 | yes (Apple silicon) | no | Intel | no | no | | HEVC Main 8-bit | yes | **yes** (hardware-only format) | QSV, NVENC, AMD | **NVENC only** | no | | HEVC Main 10 / HDR (PQ, HLG) | yes | **no** (tone-mapped to 8-bit SDR) | yes | **NVENC yes** (HDR10 SEI, `mdcv` / `clli`) | no | diff --git a/docs/performance.md b/docs/performance.md --- a/docs/performance.md +++ b/docs/performance.md @@ -80,6 +80,74 @@ What remains per live decoder is unchanged: the H.264 decoder publishes each mac picture as its own allocations (`Frame::make_row`, five per row), which is where the 290 MB per decoder comes from. +## Results (VideoToolbox export takes the RGBA pictures as they are, #3030, before → after) + +Export ▸ Hardware encoding on macOS handed VideoToolbox NV12 planes converted on the CPU by the +encode thread (`rgba_to_yuv420_8`), and that conversion, not the hardware encode, was the export's +critical path: the encode thread is the one serial thread of an export, its `par_chunks_mut` +conversion is injected into the rayon pool while the pool renders the next batch, and so a 1080p +conversion took 20–27 ms a frame (5 ms with an idle pool). `perf.stats` `export.stages` of a +60 s export: `convertMs` 29.6 s of `encodeMs` 30.3 s, with `renderMs` 30.6 s and `waitMs` 0.1 s +(encode-bound), the media engine idle most of the time, CPU 570 %, GPU 1 %. Now the encoder +takes the export's RGBA pictures in `32BGRA` buffers tagged BT.709 and VideoToolbox converts +them, as NVENC has since #486; the encode thread copies 8 MB a frame (0.3–0.4 ms) and waits for +the render (`waitMs` 18–25 s: render-bound). A VideoToolbox that refuses RGB input on the first +picture gets the NV12 path back, so the output never depends on it. + +Apple M2 Max (12 cores, 64 GB, macOS 26.6), 2026-10-23, a shared machine with other agents' +builds running (1-minute load average 57–170 across the runs, noted per pair). The timeline is +the #3030 shape: 60 s 1080p23.976 sequence, a 1080p H.264 clip on V1, a 4K H.264 clip on V2 as a +40 % picture-in-picture, stereo WAV on A1 (ffmpeg `testsrc2` + noise fixtures), exported to +H.264 20 Mb/s + AAC through `filmcraft-cli … run`, hardware decoding on. Before / after binaries +alternated (A/B/A/B/A/B), median of three; `convert`, `encode`, `render`, `wait` are the export's +own stage timers (render overlaps encode, so they do not add up to the wall time). + +| Export ▸ Hardware encoding on, CPU render | before (3 runs) | **after** (3 runs) | +|---|---|---| +| wall, median | 35.8 s (33.7 / 35.8 / 44.9) | **29.4 s** (26.8 / 29.4 / 35.5) | +| encode thread: convert | 29–39 s (20–27 ms a frame) | **0.46–0.59 s** (0.3–0.4 ms a frame) | +| encode thread: whole encode stage | 30–40 s | **1.9–6.6 s** | +| wait for the render | 0.05–0.8 s | 18–25 s | +| render (parallel, overlapped) | 30–40 s | 23–35 s | +| CPU, mean (process) | 433–566 % | 497–604 % over a shorter run (CPU-seconds 191–194 → 158–178) | +| output | 1439 frames, 148.8 MB | 1439 frames, 149.6 MB | + +Per pair (before → after, 1-minute load at the start): 44.9 → 26.8 s (57 → 80), 35.8 → 35.5 s +(61 → 67), 33.7 → 29.4 s (120 → 148); the second pair ran into a load spike during the "after" +run (67 → 120). The wall time now follows the CPU render alone. The same export with the +built-in encoder is untouched: the same bytes before and after (MD5 `12f18e96…`, 86 and 91 s at +load 54–182). + +| Export ▸ GPU Rendering on, Hardware encoding on | before (3 runs) | **after** (3 runs) | +|---|---|---| +| wall, median | 22.2 s (18.4 / 22.2 / 23.6) | **18.7 s** (18.5 / 18.7 / 19.7) | +| encode thread: convert | 6.3–11.8 s (4–8 ms a frame: the pool is freer) | **0.46–0.47 s** | +| wait for the render | 7.5–9.1 s | 14–15 s | +| CPU, mean (process) | 241–289 % | **216–219 %** | +| GPU, mean (`ioreg` device utilization) | 12–14 % | 12–14 % | + +Load 87–137 throughout, steadier than the CPU-render rows. Here too the export is now bound by +the render (GPU plan + read-back + the CPU's 8-bit conversion), which #2119 and #1284 address. + +Output check: both hardware files decode with ffmpeg without a complaint, 1439 frames, video +60.018 s / audio 60.032 s like the software export, BT.709 limited range tagged. Against the +planar path's output, VideoToolbox's own conversion gives the same luma within encoder noise +(Y 46.8 dB; the planar path against the software export is 46.1 dB) and a lower chroma PSNR on +this hard-edged synthetic pattern (Cb 38.8, Cr 35.9 dB; bias 0.1–0.2 codes, 2.8 % of chroma +samples more than 4 codes apart, all at the colour-bar and checkerboard edges): VideoToolbox +filters chroma where our conversion box-averages 2×2, not a shift (a one-sample shift of either +path scores 24–29 dB) and not a range or matrix difference. On smooth content (a 5 s +`mandelbrot` 1080p over a 4K `gradients` picture-in-picture) the two paths are 49.9 / 47.0 / +47.3 dB apart, encoder noise. `rgba_pictures_encode_like_the_cpu_conversion` holds both paths to +> 30 dB against our conversion in every plane and < 2 codes apart on average (measured 52 / 41 / +37 dB and 0.2 / 0.6 / 0.8 codes). + +Sessions: under LLDB with auto-continuing breakpoints on `VTCompressionSessionCreate` and +`VTCompressionSessionInvalidate`, two consecutive exports in one `filmcraft-cli run` created 3 +sessions (the HEVC availability probe and one per export) and invalidated every one (6 hits: ours +and the framework's at release); the test that exports at other frame rates, cancels an export +from another thread and exports again created 6 and invalidated 12. + ## Results (CPU compositor: layer rectangles, an opaque bottom layer and recycled images, before → after) The CPU compositor is what export runs. In a profile of a 9:29 export it took about a third of the