diff --git a/Cargo.lock b/Cargo.lock --- a/Cargo.lock +++ b/Cargo.lock @@ -7486,8 +7486,11 @@ version = "0.14.0" dependencies = [ "brotli", "flate2", + "lightcraft-faces", "serde_json", "sha2", + "zune-core", + "zune-jpeg", ] [[package]] diff --git a/docs/faces.md b/docs/faces.md --- a/docs/faces.md +++ b/docs/faces.md @@ -235,6 +235,67 @@ The detector's output matches OpenCV's own YuNet on a 45-photo public-domain tes mean box overlap 0.97), including marble busts and paintings, with no false boxes on the landscape and architecture photos in the set. +### And how well does it find faces? + +`cargo xtask ai-bench` measures the detector on the **WIDER FACE validation set** (3,226 photos, 39,708 faces) and prints the +average precision (AP) for the benchmark's three settings: easy, medium and hard (hard includes the smallest faces). The photos are the publisher's, not ours: they are licensed CC BY-NC-ND 4.0, for non-commercial research +use only, so the command downloads them to your machine only when you pass `--accept-terms`, and they are never part of +this repository. + +```text +cargo xtask ai-bench fetch --accept-terms --set wider-face-val # WIDER_val.zip, the evaluation toolkit's ground truth + # and YuNet, into corpus/ai-bench (SHA-256 checked; needs curl and tar or unzip) +cargo xtask ai-bench run face-detection --set wider-face-val +``` + +`--dir DIR` puts the files elsewhere (the set goes in `DIR/wider-face-val`), `--model FILE` measures another +YuNet-architecture `.onnx`, and `--limit N` scores N photos spread evenly, for a quick look. Detection uses the same +code as **Photo ▸ Detect Faces** (a 640-pixel input), keeps every box scoring at least 0.02 as the protocol expects, and +scores by the benchmark's published procedure: intersection over union of at least 0.5, scores normalised over the whole +set, 1,000 thresholds. The scoring is our own implementation of that description, not the toolkit's MATLAB code. + +Measured with the pinned YuNet 2023mar model on an Intel Core i9-13950HX (32 threads, CPU only): **87.8 easy, 83.7 medium, +59.8 hard** AP in 41 s for the 3,226 photos. The published figures for this model are 88.44 / 86.56 / 75.03. The gap is widest on hard, which includes the smallest +faces, and those are what a 640-pixel input resolves worst (the limit noted above). + +### And how well does it recognise them? + +`cargo xtask ai-bench run face-recognition --set lfw` measures the recognition models LightCraft knows by hash (SFace and +AuraFace) on **LFW** (Labeled Faces in the Wild), the standard face-verification benchmark: 6,000 pairs of photos in ten +folds, half of them the same person. The photos are the University of Massachusetts Amherst's +([vis-www.cs.umass.edu/lfw](http://vis-www.cs.umass.edu/lfw/), read through the +[Internet Archive's copy of the page](https://web.archive.org/web/20191205130041/http://vis-www.cs.umass.edu/lfw/)): news +photos of public figures, published with no licence but for research on face verification only, and the publisher warns +that no result on them shows an algorithm to be suitable for commercial use. Like WIDER FACE they go to your machine only +when you pass `--accept-terms` and are never part of this repository. The publisher's own host no longer resolves, so +`fetch` downloads the Internet Archive's copies of its `lfw.tgz` (173 MB; the md5 it prints, `a17d05bd...`, matches) and +`pairs.txt`, pinned by SHA-256. + +```text +cargo xtask ai-bench fetch --accept-terms --set lfw # lfw.tgz, pairs.txt, YuNet, SFace and AuraFace (about 470 MB) +cargo xtask ai-bench run face-recognition --set lfw +``` + +Each photo goes through the code **Photo ▸ Detect Faces** and the recognition scan use: YuNet at its default options finds the +faces, the one nearest the centre (LFW photos are centred on the person) is aligned to the ArcFace template from the +detector's five landmarks, and the model's embedding of it is compared by cosine. A photo where YuNet finds no face has its +centre cut out instead, and the count is printed (none were needed). A pair scores the cosine of its two embeddings; for each +fold the best cut-off on the other nine is applied to it, and the result is the mean accuracy over the ten folds. There is +no mirrored-face (flip) test, as the app does none. `--model FILE` measures one recognition `.onnx` LightCraft knows by its +SHA-256, and `--limit N` scores N pairs spread evenly (at least 20, so each of the ten folds has a same-person and a +different-person pair). A pair with a photo that could not be read or embedded gets no score: it is left out when the +cut-off is chosen, counts as wrong in its fold, and is reported. + +Measured on an Intel Core i9-13950HX (32 threads, CPU only: LightCraft's pure-Rust engine), all 6,000 pairs of 7,701 photos: + +| Model | Accuracy (10 folds) | Published | Wall time | +| --- | --- | --- | --- | +| SFace 2021dec | **99.47 %** (std 0.44) | 99.40 % ([OpenCV Zoo](https://github.com/opencv/opencv_zoo/blob/25f423d0e04c31a17254620e58febd7386da523b/models/face_recognition_sface/README.md), measured with its [LFW evaluation](https://github.com/opencv/opencv_zoo/blob/25f423d0e04c31a17254620e58febd7386da523b/tools/eval/README.md)) | 148 s | +| AuraFace v1 | **99.40 %** (std 0.42) | 99.65 % ([model card](https://huggingface.co/fal/AuraFace-v1/blob/af6d057c9b0ec4071d4c49c80e3539258798b609/README.md)) | 395 s | + +SFace matches its published figure. AuraFace is a quarter of a point (about 15 pairs) short of its card, which does not say +how it was aligned or whether it used a flip test; we have not tried to close that gap. + ## Suggesting who is in a photo With **Recognise faces** switched on in Settings ▸ Faces and a recognition model chosen (**Use**; the switch cannot be diff --git a/xtask/Cargo.toml b/xtask/Cargo.toml --- a/xtask/Cargo.toml +++ b/xtask/Cargo.toml @@ -10,6 +10,10 @@ workspace = true [dependencies] serde_json = { workspace = true } +# `cargo xtask ai-bench` (pure Rust): LightCraft's own face detector and JPEG decoder, both already in the lockfile +lightcraft-faces = { workspace = true } +zune-jpeg = { workspace = true } +zune-core = { workspace = true } # corpus download checksums sha2 = { workspace = true } # precompressed web bundle (`cargo xtask web`); pure Rust diff --git a/xtask/src/ai_bench/fetch.rs b/xtask/src/ai_bench/fetch.rs new file mode 100644 --- /dev/null +++ b/xtask/src/ai_bench/fetch.rs @@ -0,0 +1,192 @@ +//! `cargo xtask ai-bench fetch`: download a benchmark set and the models it needs into the benchmark folder, check each +//! file against its pinned SHA-256 and unpack the archives. Needs `curl` and `tar` (or `unzip` for a zip), as +//! `cargo xtask corpus` needs `curl`. + +use std::path::Path; +use std::process::Command; + +use lightcraft_faces::known; + +/// A file to download: name, URL, SHA-256 and whether it is an archive to unpack. +struct File { + name: &'static str, + url: &'static str, + sha256: &'static str, + archive: bool, +} + +/// A benchmark set: its folder name, the terms the user accepts by passing `--accept-terms`, its files and the models +/// (by id in `lightcraft_faces::known`) its benchmark runs. +pub struct Set { + pub id: &'static str, + terms: &'static str, + files: &'static [File], + pub models: &'static [&'static str], +} + +impl Set { + /// Each file's name and whether it is an archive (unpacked, with a completion marker). + #[cfg(test)] + pub fn archives(&self) -> Vec<(&'static str, bool)> { + self.files.iter().map(|f| (f.name, f.archive)).collect() + } +} + +pub const WIDER: Set = Set { + id: "wider-face-val", + terms: "WIDER FACE (http://shuoyang1213.me/WIDERFACE/) is licensed CC BY-NC-ND 4.0 (as its publisher's Hugging Face card states): \ +non-commercial research use only, no derivatives. The photos are downloaded to your machine and used only to measure detection accuracy; they \ +are never committed or redistributed.", + files: &[ + File { + name: "WIDER_val.zip", + url: "https://huggingface.co/datasets/CUHK-CSE/wider_face/resolve/main/data/WIDER_val.zip", + sha256: "f9efbd09f28c5d2d884be8c0eaef3967158c866a593fc36ab0413e4b2a58a17a", + archive: true, + }, + File { + name: "eval_tools.zip", + url: "http://shuoyang1213.me/WIDERFACE/support/eval_script/eval_tools.zip", + sha256: "1cf49c8243fa8a1632efe7f6aa09fd7a3994ac838c9008bdbc9dd1d547bb234a", + archive: true, + }, + ], + models: &[known::YUNET_ID], +}; + +/// The publisher's own host (vis-www.cs.umass.edu) no longer resolves, so the Internet Archive's copies of its +/// `lfw.tgz` and `pairs.txt` are used; the SHA-256 sums pin them, and `lfw.tgz` has the md5 the publisher printed. +pub const LFW: Set = Set { + id: "lfw", + terms: "LFW (Labeled Faces in the Wild, http://vis-www.cs.umass.edu/lfw/, University of Massachusetts Amherst) is a research benchmark \ +of news photos of public figures. Its publisher gives no licence but warns that it is for research on face verification and \ +must not be used to conclude that an algorithm is suitable for any commercial purpose. The photos are downloaded to your machine \ +and used only to measure recognition accuracy; they are never committed or redistributed.", + files: &[ + File { + name: "lfw.tgz", + url: "https://web.archive.org/web/20200310174752id_/http://vis-www.cs.umass.edu/lfw/lfw.tgz", + sha256: "055f7d9c632d7370e6fb4afc7468d40f970c34a80d4c6f50ffec63f5a8d536c0", + archive: true, + }, + File { + name: "pairs.txt", + url: "https://web.archive.org/web/20191202153930id_/http://vis-www.cs.umass.edu/lfw/pairs.txt", + sha256: "ea42330c62c92989f9d7c03237ed5d591365e89b3e649747777b70e692dc1592", + archive: false, + }, + ], + models: &[known::YUNET_ID, "sface-2021dec", "auraface-v1"], +}; + +/// `curl` `url` to `dest`, unless `dest` already has `sha256`; a download with another hash is deleted and an error. +fn download(url: &str, dest: &Path, sha256: &str) -> Result<(), String> { + if crate::corpus::file_sha256(dest).is_ok_and(|h| h == sha256) { + return Ok(()); + } + let mut curl = Command::new("curl"); + curl.args(["-fsSL", "--create-dirs", "-o"]).arg(dest).arg(url); + crate::run(curl, &format!("curl {url}"))?; + let actual = crate::corpus::file_sha256(dest)?; + if actual != sha256 { + std::fs::remove_file(dest).map_err(|e| format!("remove {}: {e}", dest.display()))?; + return Err(format!("{}: sha256 is {actual}, expected {sha256} (file deleted; source {url})", dest.display())); + } + Ok(()) +} + +/// Unpack `archive` (a zip or a gzipped tar) into `into` with the system's bsdtar (Windows, macOS), else `unzip`. +fn unpack(archive: &Path, into: &Path) -> Result<(), String> { + std::fs::create_dir_all(into).map_err(|e| format!("create {}: {e}", into.display()))?; + let mut tar = Command::new("tar"); + tar.arg("-xf").arg(archive).arg("-C").arg(into); + if crate::run(tar, &format!("tar -xf {}", archive.display())).is_ok() { + return Ok(()); + } + let mut unzip = Command::new("unzip"); + unzip.args(["-q", "-o"]).arg(archive).arg("-d").arg(into); + crate::run(unzip, &format!("unzip {}", archive.display())) +} + +/// Unpack `archive` (already checked against `sha256`) with `unpack` unless a completion marker for exactly that archive +/// exists. The marker is written, atomically, only after `unpack` succeeds: the unpacked folders can exist half-filled after an +/// interrupted extraction, so they never count as done and the next fetch extracts again over them. +fn unpack_once(set: &Path, archive: &str, sha256: &str, unpack: impl FnOnce() -> Result<(), String>) -> Result<(), String> { + let marker = set.join(format!(".{archive}.unpacked")); + if std::fs::read_to_string(&marker).is_ok_and(|done| done.trim() == sha256) { + return Ok(()); + } + unpack()?; + let pending = set.join(format!(".{archive}.unpacked.tmp")); + std::fs::write(&pending, sha256).map_err(|e| format!("write {}: {e}", pending.display()))?; + std::fs::rename(&pending, &marker).map_err(|e| format!("rename {}: {e}", pending.display())) +} + +/// Fetch everything the benchmarks of `sets` need under `dir`: each set's folder and `models/`. +pub fn run(dir: &Path, accept_terms: bool, sets: &[&Set]) -> Result<(), String> { + if !accept_terms { + let terms: Vec<&str> = sets.iter().map(|s| s.terms).collect(); + return Err(format!("{}\n\nPass --accept-terms to confirm you accept these terms.", terms.join("\n\n"))); + } + for set in sets { + let folder = dir.join(set.id); + for file in set.files { + download(file.url, &folder.join(file.name), file.sha256)?; + if file.archive { + unpack_once(&folder, file.name, file.sha256, || unpack(&folder.join(file.name), &folder))?; + } + } + for id in set.models { + let model = known::download(id).ok_or_else(|| format!("LightCraft offers no download of {id}"))?; + download(&model.url, &dir.join("models").join(&model.file_name), &model.sha256)?; + } + } + println!("fetched into {}", dir.display()); + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn an_interrupted_extraction_is_redone_and_a_finished_one_is_not() { + let set = std::env::temp_dir().join(format!("ai-bench-fetch-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&set); + assert!(std::fs::create_dir_all(&set).is_ok()); + // the extraction dies after creating the folder that used to count as "done" + let interrupted = unpack_once(&set, "a.zip", "abc", || { + std::fs::create_dir_all(set.join("WIDER_val").join("images")).map_err(|e| e.to_string())?; + Err("interrupted".to_string()) + }); + assert!(interrupted.is_err()); + assert!(set.join("WIDER_val").join("images").is_dir()); + let mut runs = 0; + assert!( + unpack_once(&set, "a.zip", "abc", || { + runs += 1; + Ok(()) + }) + .is_ok() + ); + assert_eq!(runs, 1, "the next fetch must extract again"); + assert!( + unpack_once(&set, "a.zip", "abc", || { + runs += 1; + Ok(()) + }) + .is_ok() + ); + assert_eq!(runs, 1, "a finished extraction is not repeated"); + // a different archive (hash) under the same name is extracted again + assert!( + unpack_once(&set, "a.zip", "def", || { + runs += 1; + Ok(()) + }) + .is_ok() + ); + assert_eq!(runs, 2); + let _ = std::fs::remove_dir_all(&set); + } +} diff --git a/xtask/src/ai_bench/lfw.rs b/xtask/src/ai_bench/lfw.rs new file mode 100644 --- /dev/null +++ b/xtask/src/ai_bench/lfw.rs @@ -0,0 +1,382 @@ +//! LFW face verification ("view 2"): 6,000 pairs of photos in ten folds, half of them the same person. Every photo goes +//! through the code Photo ▸ Detect/Recognise Faces uses (the YuNet detector, alignment to the ArcFace template from its +//! landmarks, the installed recognition model) and a pair scores the cosine of its two embeddings. + +use std::collections::HashMap; +use std::io::Read; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lightcraft_faces::align::{Rgb, align_to_template, crop_box}; +use lightcraft_faces::matching::cosine; +use lightcraft_faces::runtime::Embedder; +use lightcraft_faces::yunet::{Detector, Options}; + +use super::metrics; +use super::wider::load_rgb; + +/// How much wider than the guessed face box the fallback crop is (what the app does for a box without landmarks). +const BOX_PAD: f32 = 1.5; +/// Half the side of the fallback box, as a fraction of the shorter side of the photo (LFW centres the face). +const FALLBACK_HALF: f32 = 0.22; + +/// Largest `pairs.txt` accepted (the real file is under 200 KB). +const MAX_PAIRS_BYTES: u64 = 1 << 20; + +/// Refuse a `limit` that leaves some fold without a same-person and a different-person pair to evaluate and train on. +fn check_limit(limit: Option, pairs: &[Pair]) -> Result<(), String> { + let folds = pairs.iter().map(|p| p.fold).max().map_or(0, |last| last.saturating_add(1)); + let minimum = folds.saturating_mul(2); + match limit { + Some(k) if k < minimum => Err(format!( + "--limit {k} is too small to evaluate {folds} folds: use at least {minimum} pairs (a same-person and a different-person pair per fold)" + )), + _ => Ok(()), + } +} + +/// One line of `pairs.txt`. +#[derive(Debug, PartialEq)] +pub struct Pair { + pub a: PathBuf, + pub b: PathBuf, + pub same: bool, + pub fold: usize, +} + +/// Read `dir/pairs.txt` (LFW's view 2) and check that every photo it names is under `dir/lfw`. The file starts with +/// ` `; each fold then lists that many same-person lines (`name n m`) followed by as +/// many different-person lines (`name n other m`). +pub fn load_pairs(dir: &Path) -> Result, String> { + let list = dir.join("pairs.txt"); + let unreadable = |e: std::io::Error| format!("{}: {e} (run `cargo xtask ai-bench fetch --set lfw`)", list.display()); + let file = std::fs::File::open(&list).map_err(unreadable)?; + let size = file.metadata().map_err(unreadable)?.len(); + if size > MAX_PAIRS_BYTES { + return Err(format!("{}: is {size} bytes, more than the {MAX_PAIRS_BYTES} a pairs file can be", list.display())); + } + // read one byte past the cap too, in case the file grows while it is read + let mut text = String::new(); + file.take(MAX_PAIRS_BYTES + 1).read_to_string(&mut text).map_err(unreadable)?; + if text.len() as u64 > MAX_PAIRS_BYTES { + return Err(format!("{}: grew past {MAX_PAIRS_BYTES} bytes while being read", list.display())); + } + let photos = dir.join("lfw"); + let photo = |name: &str, number: &str| -> Result { + let number: usize = number.parse().map_err(|_| format!("{}: `{number}` is not a photo number", list.display()))?; + if name.is_empty() || name.contains(['/', '\\', ':']) || name.contains("..") { + return Err(format!("{}: `{name}` is not a person's folder name", list.display())); + } + Ok(photos.join(name).join(format!("{name}_{number:04}.jpg"))) + }; + let mut lines = text.lines(); + let header: Vec = lines.next().unwrap_or("").split_whitespace().map_while(|t| t.parse().ok()).collect(); + let [folds, half] = header.as_slice() else { + return Err(format!("{}: the first line must be ` `", list.display())); + }; + // the header is data from the file: a damaged one must not overflow + let too_big = || format!("{}: the header `{folds} {half}` is too large", list.display()); + let per_fold = half.checked_mul(2).filter(|n| *n > 0).ok_or_else(too_big)?; + let expected = folds.checked_mul(per_fold).ok_or_else(too_big)?; + let mut pairs = Vec::new(); + for (i, line) in lines.filter(|l| !l.trim().is_empty()).enumerate() { + let fold = i / per_fold; + let words: Vec<&str> = line.split_whitespace().collect(); + let (a, b, same) = match words.as_slice() { + [name, n, m] => (photo(name, n)?, photo(name, m)?, true), + [name, n, other, m] => (photo(name, n)?, photo(other, m)?, false), + _ => return Err(format!("{}: pair {} is malformed: `{line}`", list.display(), i + 1)), + }; + if same != (i % per_fold < *half) { + let (place, kind) = if same { ("different-person", "three fields") } else { ("same-person", "four fields") }; + return Err(format!("{}: pair {} is in the {place} half of fold {} but has {kind}: `{line}`", list.display(), i + 1, fold + 1)); + } + pairs.push(Pair { a, b, same, fold }); + } + if pairs.len() != expected { + return Err(format!("{}: lists {} pairs, its header says {expected}", list.display(), pairs.len())); + } + if let Some(missing) = pairs.iter().flat_map(|p| [&p.a, &p.b]).find(|p| !p.is_file()) { + return Err(format!("{}: no such photo (run `cargo xtask ai-bench fetch --set lfw`)", missing.display())); + } + Ok(pairs) +} + +/// Run `work(0..n)` on `threads` threads and give the results in order. +fn parallel(n: usize, threads: usize, work: impl Fn(usize) -> T + Sync) -> Result, String> { + let next = AtomicUsize::new(0); + let finished = AtomicUsize::new(0); + let started = Instant::now(); + let step = (n / 20).max(1); + let joined = std::thread::scope(|scope| { + let workers: Vec<_> = (0..threads.clamp(1, 64)) + .map(|_| { + scope.spawn(|| { + let mut done = Vec::new(); + loop { + let at = next.fetch_add(1, Ordering::Relaxed); + if at >= n { + break; + } + done.push((at, work(at))); + let count = finished.fetch_add(1, Ordering::Relaxed) + 1; + if count.is_multiple_of(step) { + eprintln!(" {count}/{n} ({:.1}/s)", count as f64 / started.elapsed().as_secs_f64().max(1e-9)); + } + } + done + }) + }) + .collect(); + workers.into_iter().map(|w| w.join().map_err(|_| "a worker stopped unexpectedly".to_string())).collect::, _>>() + })?; + let mut all: Vec<(usize, T)> = joined.into_iter().flatten().collect(); + all.sort_by_key(|(at, _)| *at); + Ok(all.into_iter().map(|(_, value)| value).collect()) +} + +/// The face of one photo as the recogniser wants it, and whether the detector found none (so the centre was cut out). +/// LFW photos are centred on the person, so with several faces the one nearest the centre is the one. +fn face(detector: &Detector, path: &Path, size: (usize, usize)) -> Result<(Vec, bool), String> { + let (rgb, w, h) = load_rgb(path)?; + let faces = detector.detect(&rgb, w, h, &Options::default()).map_err(|e| format!("{}: {e}", path.display()))?; + let picture = Rgb { data: &rgb, width: w, height: h }; + let (fw, fh) = (w as f32, h as f32); + let distance = |f: &lightcraft_faces::yunet::Face| ((f.x0 + f.x1) / 2.0 - 0.5).hypot((f.y0 + f.y1) / 2.0 - 0.5); + let aligned = faces + .iter() + .min_by(|a, b| distance(a).total_cmp(&distance(b))) + .and_then(|f| align_to_template(&picture, &f.landmarks.map(|(x, y)| (x * fw, y * fh)), size.0, size.1)); + if let Some(crop) = aligned { + return Ok((crop, false)); + } + let half = fw.min(fh) * FALLBACK_HALF; + let crop = crop_box(&picture, fw / 2.0 - half, fh / 2.0 - half, fw / 2.0 + half, fh / 2.0 + half, BOX_PAD, size.0, size.1) + .ok_or_else(|| format!("{}: no face could be cut out", path.display()))?; + Ok((crop, true)) +} + +/// A pair's score: the cosine of its two photos' embeddings; a pair with a photo that could not be embedded (or whose +/// embeddings cannot be compared) has none, so it is counted as wrong instead of being given a made-up score. +fn scores(pairs: &[Pair], embeddings: &HashMap<&Path, Vec>) -> Vec<(Option, bool, usize)> { + pairs + .iter() + .map(|p| { + let score = match (embeddings.get(p.a.as_path()), embeddings.get(p.b.as_path())) { + (Some(a), Some(b)) => cosine(a, b).map(f64::from), + _ => None, + }; + (score, p.same, p.fold) + }) + .collect() +} + +pub struct Run { + pub pairs: usize, + pub photos: usize, + /// Photos where the detector found no face and the centre was cut out. + pub fallbacks: usize, + /// Photos that could not be read or embedded. + pub failed: usize, + /// Pairs with such a photo: they have no score, take no part in choosing a cut-off and count as wrong. + pub failed_pairs: usize, + pub seconds: f64, + /// Mean and standard deviation of the ten folds' accuracy, in percent. + pub accuracy: (f64, f64), +} + +/// Verify `embedder` on the pairs under `dir` (or `limit` of them, spread evenly) on `threads` threads. +pub fn run(dir: &Path, detector: &Detector, embedder: &Embedder, limit: Option, threads: usize) -> Result { + let mut pairs = load_pairs(dir)?; + check_limit(limit, &pairs)?; + if let Some(k) = limit.filter(|k| *k > 0 && *k < pairs.len()) { + let n = pairs.len(); + let keep: Vec = (0..k).map(|i| i * n / k).collect(); + pairs = pairs.into_iter().enumerate().filter(|(i, _)| keep.contains(i)).map(|(_, p)| p).collect(); + } + let mut photos: Vec<&Path> = pairs.iter().flat_map(|p| [p.a.as_path(), p.b.as_path()]).collect(); + photos.sort(); + photos.dedup(); + let started = Instant::now(); + let size = embedder.input_size(); + let found = parallel(photos.len(), threads, |i| { + let path = photos.get(i).ok_or("no such photo")?; + let (crop, fallback) = face(detector, path, size)?; + let embedding = embedder.embed(&crop).map_err(|e| format!("{}: {e}", path.display()))?; + Ok::<_, String>((embedding, fallback)) + })?; + let (mut embeddings, mut fallbacks, mut failed) = (HashMap::new(), 0, 0); + for (path, result) in photos.iter().zip(found) { + match result { + Ok((embedding, fallback)) => { + fallbacks += usize::from(fallback); + embeddings.insert(*path, embedding); + } + Err(e) => { + if failed < 5 { + eprintln!(" warning: {e}"); + } + failed += 1; + } + } + } + let scored = scores(&pairs, &embeddings); + let failed_pairs = scored.iter().filter(|p| p.0.is_none()).count(); + let (mean, std) = metrics::fold_accuracy(&scored); + Ok(Run { + pairs: pairs.len(), + failed_pairs, + photos: photos.len(), + fallbacks, + failed, + seconds: started.elapsed().as_secs_f64(), + accuracy: (mean * 100.0, std * 100.0), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A scratch directory that is removed afterwards. + struct Scratch(PathBuf); + + impl Scratch { + fn new(name: &str) -> Scratch { + let dir = std::env::temp_dir().join(format!("ai-bench-lfw-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + assert!(std::fs::create_dir_all(&dir).is_ok()); + Scratch(dir) + } + + fn pairs(&self, text: &str) { + assert!(std::fs::write(self.0.join("pairs.txt"), text).is_ok()); + } + + fn photo(&self, name: &str, number: usize) { + let folder = self.0.join("lfw").join(name); + assert!(std::fs::create_dir_all(&folder).is_ok()); + assert!(std::fs::write(folder.join(format!("{name}_{number:04}.jpg")), b"x").is_ok()); + } + } + + impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } + } + + /// Two folds of one same-person and one different-person pair each. + const TWO_FOLDS: &str = "2 1\nAnn\t1\t2\nAnn\t1\tBob\t1\nBob\t1\t2\nBob\t2\tAnn\t2\n"; + + fn two_folds() -> Scratch { + let s = Scratch::new(&format!("two-{:?}", std::thread::current().id()).replace(['(', ')'], "")); + for (name, n) in [("Ann", 1), ("Ann", 2), ("Bob", 1), ("Bob", 2)] { + s.photo(name, n); + } + s.pairs(TWO_FOLDS); + s + } + + #[test] + fn pairs_read_into_folds_of_same_then_different() { + let s = two_folds(); + let pairs = load_pairs(&s.0).unwrap(); + let lfw = s.0.join("lfw"); + assert_eq!(pairs.len(), 4); + assert_eq!(pairs[0], Pair { a: lfw.join("Ann/Ann_0001.jpg"), b: lfw.join("Ann/Ann_0002.jpg"), same: true, fold: 0 }); + assert_eq!(pairs[1], Pair { a: lfw.join("Ann/Ann_0001.jpg"), b: lfw.join("Bob/Bob_0001.jpg"), same: false, fold: 0 }); + assert_eq!((pairs[2].same, pairs[2].fold, pairs[3].same, pairs[3].fold), (true, 1, false, 1)); + } + + #[test] + fn a_pairs_file_over_the_size_cap_is_refused_with_its_size() { + let s = Scratch::new("big"); + let size = (MAX_PAIRS_BYTES + 1) as usize; + s.pairs(&" ".repeat(size)); + let e = load_pairs(&s.0).err().unwrap_or_default(); + assert!(e.contains(&size.to_string()) && e.contains(&MAX_PAIRS_BYTES.to_string()), "{e:?}"); + } + + #[test] + fn a_pair_of_the_wrong_kind_for_its_place_in_the_fold_is_an_error() { + let s = two_folds(); + for (text, why) in [ + ("2 1\nAnn 1 Bob 1\nAnn 1 2\nBob 1 2\nBob 2 Ann 2\n", "different-person pair in the same-person half"), + ("2 1\nAnn 1 2\nAnn 1 2\nBob 1 2\nBob 2 Ann 2\n", "same-person pair in the different-person half"), + ("2 1\nAnn 1 2\nAnn 1 Bob 1\nBob 2 Ann 2\nBob 1 2\n", "halves swapped in the second fold"), + ] { + s.pairs(text); + let e = load_pairs(&s.0).err().unwrap_or_default(); + assert!(e.contains("pair "), "{why}: {e:?}"); + } + } + + #[test] + fn a_limit_too_small_for_ten_fold_evaluation_is_refused_and_states_the_minimum() { + let s = Scratch::new("limit"); + let pair = |fold| Pair { a: s.0.join("a"), b: s.0.join("b"), same: true, fold }; + let pairs: Vec = (0..10).map(pair).collect(); + for k in [1, 5, 19] { + let e = check_limit(Some(k), &pairs).err().unwrap_or_default(); + assert!(e.contains("20") && e.contains("10 folds"), "{k}: {e:?}"); + } + assert!(check_limit(Some(20), &pairs).is_ok() && check_limit(None, &pairs).is_ok()); + } + + #[test] + fn a_missing_pairs_file_or_photo_is_an_error_that_says_how_to_fetch_it() { + let s = Scratch::new("missing"); + assert!(load_pairs(&s.0).err().unwrap().contains("ai-bench fetch")); + s.pairs(TWO_FOLDS); + s.photo("Ann", 1); + let e = load_pairs(&s.0).err().unwrap(); + assert!(e.contains("Ann_0002.jpg") && e.contains("ai-bench fetch"), "{e}"); + } + + #[test] + fn malformed_pairs_files_are_errors_not_panics() { + let s = two_folds(); + for (text, why) in [ + ("", "empty"), + ("garbage\n", "no header"), + ("2\nAnn 1 2\n", "one-number header"), + ("18446744073709551615 2\nAnn 1 2\n", "a header that overflows the pair count"), + ("2 9223372036854775807\nAnn 1 2\n", "a header that overflows the fold size"), + ("0 0\n", "empty folds"), + ("2 1\nAnn 1 2\nAnn 1 Bob 1\nBob 1 2\n", "a pair short"), + ("2 1\nAnn 1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\nAnn 1 2\n", "a pair too many"), + ("2 1\nAnn 1\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a line of two words"), + ("2 1\nAnn x 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a photo number that is not a number"), + ("2 1\nAnn -1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a negative photo number"), + ("2 1\n../Ann 1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a name that climbs out"), + ("2 1\nC:Ann 1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a drive in the name"), + ] { + s.pairs(text); + assert!(load_pairs(&s.0).is_err(), "{why}"); + } + // a pairs file that is not text + assert!(std::fs::write(s.0.join("pairs.txt"), [0xff, 0xfe, 0x00]).is_ok()); + assert!(load_pairs(&s.0).is_err()); + } + + #[test] + fn a_pair_scores_the_cosine_of_its_embeddings_and_a_missing_one_scores_lowest() { + let s = two_folds(); + let pairs = load_pairs(&s.0).unwrap(); + let lfw = s.0.join("lfw"); + let path = |n: &str| lfw.join(n.split('_').next().unwrap_or("")).join(format!("{n}.jpg")); + let (a1, a2, b1) = (path("Ann_0001"), path("Ann_0002"), path("Bob_0001")); + let mut embeddings: HashMap<&Path, Vec> = HashMap::new(); + embeddings.insert(a1.as_path(), vec![1.0, 0.0]); + embeddings.insert(a2.as_path(), vec![0.6, 0.8]); + embeddings.insert(b1.as_path(), vec![0.0, 1.0]); + let scored = scores(&pairs, &embeddings); + assert_eq!(scored.len(), 4); + assert!((scored[0].0.unwrap_or(f64::NAN) - 0.6).abs() < 1e-6 && scored[0].1 && scored[0].2 == 0); + assert!(scored[1].0.unwrap_or(f64::NAN).abs() < 1e-6 && !scored[1].1); + // Bob_0002 has no embedding: no score at all, not a low one + assert_eq!((scored[2].0, scored[3].0), (None, None)); + } +} diff --git a/xtask/src/ai_bench/mat.rs b/xtask/src/ai_bench/mat.rs new file mode 100644 --- /dev/null +++ b/xtask/src/ai_bench/mat.rs @@ -0,0 +1,315 @@ +//! A reader for MATLAB level-5 MAT-files (the format of the WIDER FACE ground truth), written from MathWorks' published +//! file-format description: numeric arrays, char arrays and cell arrays, compressed or not, little-endian. Sizes are +//! checked against the data, nesting is bounded, and anything unexpected is an error. + +use std::io::Read; + +const MAX_DEPTH: usize = 16; +/// Largest decompressed variable. +const MAX_INFLATED: u64 = 1 << 30; +/// Largest single dimension of a matrix (every real variable is far smaller; a hostile one must not size an allocation). +const MAX_DIM: usize = 1 << 30; + +#[derive(Clone, Debug, PartialEq)] +pub enum Value { + /// A numeric matrix, column-major, as f64. + Num { + dims: Vec, + data: Vec, + }, + Char(String), + Cell(Vec), + /// Any other class (struct, sparse, objects): present but not read. + Other, +} + +impl Value { + pub fn cells(&self) -> Option<&[Value]> { + match self { + Value::Cell(items) => Some(items), + _ => None, + } + } + + pub fn text(&self) -> Option<&str> { + match self { + Value::Char(text) => Some(text), + _ => None, + } + } + + /// Rows of a numeric matrix (stored column-major); an empty matrix is no rows. + pub fn rows(&self) -> Option>> { + let Value::Num { dims, data } = self else { return None }; + let (r, c) = (dims.first().copied().unwrap_or(0), dims.get(1).copied().unwrap_or(1)); + if r.checked_mul(c) != Some(data.len()) { + return None; + } + if data.is_empty() { + return Some(Vec::new()); + } + Some((0..r).map(|i| (0..c).filter_map(|j| data.get(i + j * r).copied()).collect()).collect()) + } + + /// All numbers of a numeric matrix. + pub fn numbers(&self) -> Option<&[f64]> { + match self { + Value::Num { data, .. } => Some(data), + _ => None, + } + } +} + +fn bad(what: &str) -> String { + format!("malformed MAT-file: {what}") +} + +struct Reader<'a> { + b: &'a [u8], + at: usize, +} + +impl<'a> Reader<'a> { + fn bytes(&self, len: usize) -> Option<&'a [u8]> { + self.b.get(self.at..self.at.checked_add(len)?) + } + + fn u32(&mut self) -> Result { + let word = self.bytes(4).and_then(|s| <[u8; 4]>::try_from(s).ok()).ok_or_else(|| bad("truncated tag"))?; + self.at += 4; + Ok(u32::from_le_bytes(word)) + } + + fn done(&self) -> bool { + self.at >= self.b.len() + } + + /// The next data element: its type and bytes. + fn element(&mut self) -> Result<(u32, &'a [u8]), String> { + let first = self.u32()?; + if first >> 16 != 0 { + // small element: type and size in one word, data in the next four bytes + let (ty, size) = (first & 0xffff, (first >> 16) as usize); + let data = self.bytes(size.min(4)).ok_or_else(|| bad("truncated small element"))?; + self.at += 4; + return Ok((ty, data)); + } + let size = self.u32()? as usize; + let data = self.bytes(size).ok_or_else(|| bad("an element runs past the end of the file"))?; + let end = self.at + size; + // compressed elements are not padded; everything else is padded to 8 bytes + self.at = if first == 15 { end } else { end.checked_next_multiple_of(8).ok_or_else(|| bad("size overflow"))? }; + Ok((first, data)) + } +} + +/// The values of a numeric data element of MAT type `ty`. +fn numbers(ty: u32, d: &[u8]) -> Result, String> { + fn le(d: &[u8]) -> impl Iterator + '_ { + d.as_chunks::().0.iter().copied() + } + Ok(match ty { + 1 => d.iter().map(|b| f64::from(*b as i8)).collect(), + 2 => d.iter().map(|b| f64::from(*b)).collect(), + 3 => le::<2>(d).map(|c| f64::from(i16::from_le_bytes(c))).collect(), + 4 => le::<2>(d).map(|c| f64::from(u16::from_le_bytes(c))).collect(), + 5 => le::<4>(d).map(|c| f64::from(i32::from_le_bytes(c))).collect(), + 6 => le::<4>(d).map(|c| f64::from(u32::from_le_bytes(c))).collect(), + 7 => le::<4>(d).map(|c| f64::from(f32::from_le_bytes(c))).collect(), + 9 => le::<8>(d).map(f64::from_le_bytes).collect(), + 12 => le::<8>(d).map(|c| i64::from_le_bytes(c) as f64).collect(), + 13 => le::<8>(d).map(|c| u64::from_le_bytes(c) as f64).collect(), + _ => return Err(bad(&format!("numeric data of type {ty}"))), + }) +} + +/// A miMATRIX element's body. +fn matrix(body: &[u8], depth: usize) -> Result<(String, Value), String> { + if depth > MAX_DEPTH { + return Err(bad("nested too deeply")); + } + if body.is_empty() { + return Ok((String::new(), Value::Num { dims: vec![0, 0], data: Vec::new() })); + } + let mut r = Reader { b: body, at: 0 }; + let (_, flags) = r.element()?; + let class = flags.first().copied().ok_or_else(|| bad("no array flags"))?; + let complex = flags.get(1).is_some_and(|f| f & 0x08 != 0); + let (dty, dims) = r.element()?; + let dims: Vec = numbers(dty, dims)?.into_iter().map(|d| if d.is_finite() && d >= 0.0 { d as usize } else { 0 }).collect(); + if dims.iter().any(|d| *d > MAX_DIM) { + return Err(bad("a dimension larger than any real variable")); + } + let count = dims.iter().try_fold(1usize, |n, d| n.checked_mul(*d)).ok_or_else(|| bad("dimensions overflow"))?; + let (_, name) = r.element()?; + let name = String::from_utf8_lossy(name).into_owned(); + let value = match class { + 1 => { + // every item takes at least a tag (8 bytes), which bounds a hostile count by the data + if count > body.len() / 8 { + return Err(bad("a cell array larger than its data")); + } + let mut items = Vec::with_capacity(count); + for _ in 0..count { + let (ty, b) = r.element()?; + if ty != 14 { + return Err(bad("a cell that is not a matrix")); + } + items.push(matrix(b, depth + 1)?.1); + } + Value::Cell(items) + } + 4 => { + let (ty, d) = r.element()?; + let text = match ty { + 16 => String::from_utf8_lossy(d).into_owned(), + _ => numbers(ty, d)?.into_iter().map(|c| char::from_u32(c as u32).unwrap_or('\u{fffd}')).collect(), + }; + Value::Char(text) + } + 6..=15 => { + let (ty, d) = r.element()?; + let data = numbers(ty, d)?; + if data.len() != count { + return Err(bad("a matrix whose data does not match its dimensions")); + } + if complex && !r.done() { + r.element()?; + } + Value::Num { dims, data } + } + _ => Value::Other, + }; + Ok((name, value)) +} + +/// The variables in a MAT-file's bytes, by name. +pub fn parse(bytes: &[u8]) -> Result, String> { + if bytes.len() < 128 { + return Err(bad("shorter than its header")); + } + if bytes.get(126..128) != Some(b"IM") { + return Err(bad("not a little-endian level-5 MAT-file (v7.3 HDF5 files are not supported)")); + } + let mut r = Reader { b: bytes, at: 128 }; + let mut out = Vec::new(); + while !r.done() { + let (ty, body) = r.element()?; + match ty { + 14 => out.push(matrix(body, 0)?), + 15 => { + let mut inflated = Vec::new(); + flate2::read::ZlibDecoder::new(body) + .take(MAX_INFLATED) + .read_to_end(&mut inflated) + .map_err(|e| bad(&format!("compressed variable: {e}")))?; + let (ity, ibody) = Reader { b: &inflated, at: 0 }.element()?; + if ity == 14 { + out.push(matrix(ibody, 0)?); + } + } + _ => {} + } + } + Ok(out) +} + +/// The variable `name` in `vars`. +pub fn var<'a>(vars: &'a [(String, Value)], name: &str) -> Result<&'a Value, String> { + vars.iter().find(|(n, _)| n == name).map(|(_, v)| v).ok_or_else(|| format!("the MAT-file has no variable `{name}`")) +} + +/// Builders for synthetic MAT-files, shared with the ground-truth tests. +#[cfg(test)] +pub(crate) mod tests { + use super::*; + + fn element(ty: u32, data: &[u8]) -> Vec { + let mut out = ty.to_le_bytes().to_vec(); + out.extend_from_slice(&(data.len() as u32).to_le_bytes()); + out.extend_from_slice(data); + out.resize(out.len().div_ceil(8) * 8, 0); + out + } + + fn header(class: u8, dims: &[i32], name: &str) -> Vec { + let mut out = element(6, &[class, 0, 0, 0, 0, 0, 0, 0]); + out.extend(element(5, &dims.iter().flat_map(|d| d.to_le_bytes()).collect::>())); + out.extend(element(1, name.as_bytes())); + out + } + + /// A double matrix as a miMATRIX element (column-major `data`). + pub(crate) fn double(name: &str, rows: i32, cols: i32, data: &[f64]) -> Vec { + let mut body = header(6, &[rows, cols], name); + body.extend(element(9, &data.iter().flat_map(|d| d.to_le_bytes()).collect::>())); + element(14, &body) + } + + pub(crate) fn chars(name: &str, text: &str) -> Vec { + let mut body = header(4, &[1, text.len() as i32], name); + body.extend(element(16, text.as_bytes())); + element(14, &body) + } + + pub(crate) fn cell(name: &str, items: &[Vec]) -> Vec { + let mut body = header(1, &[items.len() as i32, 1], name); + for item in items { + body.extend_from_slice(item); + } + element(14, &body) + } + + /// A MAT-file: the 128-byte header, then `variables`. + pub(crate) fn file(variables: &[Vec]) -> Vec { + let mut out = vec![0u8; 128]; + out[126] = b'I'; + out[127] = b'M'; + for v in variables { + out.extend_from_slice(v); + } + out + } + + #[test] + fn reads_numbers_text_and_nested_cells() { + let bytes = file(&[double("m", 2, 2, &[1.0, 2.0, 3.0, 4.0]), cell("c", &[chars("", "ab"), cell("", &[double("", 1, 1, &[7.0])])])]); + let vars = parse(&bytes).unwrap(); + // column-major [1 3; 2 4] has the rows [1, 3] and [2, 4] + assert_eq!(var(&vars, "m").unwrap().rows(), Some(vec![vec![1.0, 3.0], vec![2.0, 4.0]])); + let c = var(&vars, "c").unwrap().cells().unwrap(); + assert_eq!(c[0].text(), Some("ab")); + assert_eq!(c[1].cells().unwrap()[0].numbers(), Some(&[7.0][..])); + assert!(var(&vars, "absent").is_err()); + } + + #[test] + fn a_matrix_without_data_has_no_rows_whatever_its_dimensions_claim() { + // dims [N, 0] hold no data; building N empty rows from a hostile N would exhaust memory + let empty = Value::Num { dims: vec![1 << 20, 0], data: Vec::new() }; + assert_eq!(empty.rows(), Some(Vec::new())); + // a dimension no real variable has is refused at parse time + let hostile = file(&[double("m", i32::MAX, 0, &[])]); + assert!(parse(&hostile).is_err()); + // an ordinary empty matrix still parses and has no rows + let vars = parse(&file(&[double("m", 0, 0, &[])])).unwrap(); + assert_eq!(var(&vars, "m").unwrap().rows(), Some(Vec::new())); + } + + #[test] + fn malformed_files_are_errors() { + assert!(parse(b"too short").is_err()); + let mut wrong_endian = file(&[]); + wrong_endian[126] = b'M'; + wrong_endian[127] = b'I'; + assert!(parse(&wrong_endian).is_err()); + let good = file(&[double("m", 1, 2, &[1.0, 2.0])]); + for cut in [130, 140, good.len() - 1] { + assert!(parse(&good[..cut]).is_err(), "cut at {cut}"); + } + // a cell array that claims far more items than it has bytes + let mut hostile = header(1, &[i32::MAX, 1], "c"); + hostile.extend(element(14, &[])); + assert!(parse(&file(&[element(14, &hostile)])).is_err()); + } +} diff --git a/xtask/src/ai_bench/metrics.rs b/xtask/src/ai_bench/metrics.rs new file mode 100644 --- /dev/null +++ b/xtask/src/ai_bench/metrics.rs @@ -0,0 +1,295 @@ +//! WIDER FACE average precision, implemented from the published description of the evaluation (Yang et al., "WIDER +//! FACE", CVPR 2016, and the evaluation toolkit's documented procedure; no evaluation code was copied): detection +//! scores are normalised over the whole set, a detection hits a face when their intersection over union is at least 0.5 +//! (with the toolkit's inclusive-pixel `+1` convention), faces outside the setting's keep list are neither counted nor +//! punished, precision and recall are taken at 1,000 score thresholds and the area under the curve is the VOC +//! all-point average precision. + +/// How many score thresholds the precision/recall curve has. +const THRESHOLDS: usize = 1000; +/// Overlap at which a detection hits a face. +const MIN_IOU: f64 = 0.5; + +/// One image: ground-truth boxes `[x, y, w, h]`, the 1-based indices of the faces this setting counts, and detections +/// `[x, y, w, h, score]`, all in pixels. +pub struct Image<'a> { + pub gt: &'a [[f64; 4]], + pub keep: &'a [usize], + pub preds: &'a [[f64; 5]], +} + +/// IoU of two `[x1, y1, x2, y2]` boxes, counting pixels inclusively (a box from 0 to 9 is ten pixels wide). +fn iou(a: &[f64; 4], b: &[f64; 4]) -> f64 { + let iw = a[2].min(b[2]) - a[0].max(b[0]) + 1.0; + let ih = a[3].min(b[3]) - a[1].max(b[1]) + 1.0; + if iw <= 0.0 || ih <= 0.0 { + return 0.0; + } + let area = |r: &[f64; 4]| (r[2] - r[0] + 1.0) * (r[3] - r[1] + 1.0); + let inter = iw * ih; + let union = area(a) + area(b) - inter; + if union > 0.0 { inter / union } else { 0.0 } +} + +/// VOC all-point average precision of a precision/recall curve (recall ascending). +fn voc_ap(recall: &[f64], precision: &[f64]) -> f64 { + let mut mrec = vec![0.0]; + mrec.extend_from_slice(recall); + mrec.push(1.0); + let mut mpre = vec![0.0]; + mpre.extend_from_slice(precision); + mpre.push(0.0); + for i in (1..mpre.len()).rev() { + mpre[i - 1] = mpre[i - 1].max(mpre[i]); + } + mrec.windows(2).zip(mpre.iter().skip(1)).filter(|(r, _)| r[1] != r[0]).map(|(r, p)| (r[1] - r[0]) * p).sum() +} + +/// The WIDER FACE average precision (0 to 1) of one setting (easy, medium or hard: the `keep` lists) over `images`. +pub fn wider_ap(images: &[Image]) -> f64 { + let (mut lo, mut hi) = (f64::INFINITY, f64::NEG_INFINITY); + for p in images.iter().flat_map(|i| i.preds).filter(|p| p[4].is_finite()) { + lo = lo.min(p[4]); + hi = hi.max(p[4]); + } + let span = if hi > lo { hi - lo } else { 1.0 }; + // per threshold: detections kept that are not on an uncounted face, and counted faces found + let mut curve = vec![(0.0f64, 0.0f64); THRESHOLDS]; + let mut faces = 0usize; + for img in images { + faces += img.keep.len(); + if img.gt.is_empty() || img.preds.is_empty() { + continue; + } + let gt: Vec<[f64; 4]> = img.gt.iter().map(|g| [g[0], g[1], g[0] + g[2], g[1] + g[3]]).collect(); + let mut preds: Vec<([f64; 4], f64)> = img + .preds + .iter() + .filter(|p| p[4].is_finite()) + .map(|p| ([p[0], p[1], p[0] + p[2], p[1] + p[3]], ((p[4] - lo) / span).clamp(0.0, 1.0))) + .collect(); + preds.sort_by(|a, b| b.1.total_cmp(&a.1)); + // per face: whether this setting counts it, and 0 unseen / 1 found / -1 matched by a detection that is then ignored + let mut state: Vec<(bool, i8)> = vec![(false, 0); gt.len()]; + for k in img.keep { + if let Some(s) = k.checked_sub(1).and_then(|i| state.get_mut(i)) { + s.0 = true; + } + } + // after each detection (best first): the proposals so far and the counted faces found so far + let (mut proposals, mut recalled) = (0usize, 0usize); + let mut after: Vec<(usize, usize)> = Vec::with_capacity(preds.len()); + for (b, _) in &preds { + let mut best = (0.0f64, None); + for (j, g) in gt.iter().enumerate() { + let o = iou(b, g); + if o > best.0 { + best = (o, Some(j)); + } + } + let mut ignored = false; + if let (o, Some(j)) = best + && o >= MIN_IOU + && let Some((counted, found)) = state.get_mut(j) + { + if !*counted { + if *found == 1 { + recalled = recalled.saturating_sub(1); + } + *found = -1; + ignored = true; + } else if *found == 0 { + *found = 1; + recalled += 1; + } + } + if !ignored { + proposals += 1; + } + after.push((proposals, recalled)); + } + for (t, point) in curve.iter_mut().enumerate() { + let threshold = 1.0 - (t + 1) as f64 / THRESHOLDS as f64; + if let Some((p, r)) = preds.iter().rposition(|p| p.1 >= threshold).and_then(|last| after.get(last)) { + point.0 += *p as f64; + point.1 += *r as f64; + } + } + } + if faces == 0 { + return 0.0; + } + let precision: Vec = curve.iter().map(|c| if c.0 > 0.0 { c.1 / c.0 } else { 0.0 }).collect(); + let recall: Vec = curve.iter().map(|c| c.1 / faces as f64).collect(); + voc_ap(&recall, &precision) +} + +/// The cut-off (a pair is "same" when its score is at least the cut-off) that gets most of `pairs` (score, same person) +/// right. Non-finite scores are ignored; among equally good cut-offs the lowest wins. +fn best_cutoff(pairs: &[(f64, bool)]) -> f64 { + let mut sorted: Vec<(f64, bool)> = pairs.iter().copied().filter(|p| p.0.is_finite()).collect(); + sorted.sort_by(|a, b| a.0.total_cmp(&b.0)); + // a cut-off below every score calls every pair "same" + let mut right = sorted.iter().filter(|p| p.1).count() as i64; + let mut best = (right, sorted.first().map_or(0.0, |p| p.0 - 1.0)); + let mut groups = sorted.chunk_by(|a, b| a.0 == b.0).peekable(); + while let Some(group) = groups.next() { + // raising the cut-off past this score turns its pairs into "different" + right += group.iter().map(|p| if p.1 { -1 } else { 1 }).sum::(); + let score = group.first().map_or(0.0, |p| p.0); + let cutoff = groups.peek().and_then(|next| next.first()).map_or(score + 1.0, |n| (score + n.0) / 2.0); + if right > best.0 { + best = (right, cutoff); + } + } + best.1 +} + +/// LFW's verification accuracy: 10-fold cross-validation over `pairs` (score, same person, fold). For each fold the +/// cut-off that is best on the other folds is applied to it; the result is the mean and the standard deviation of the +/// folds' accuracies, as fractions. A pair without a score (`None`: a photo could not be embedded) takes no part in +/// choosing a cut-off and counts as wrong in its own fold. +pub fn fold_accuracy(pairs: &[(Option, bool, usize)]) -> (f64, f64) { + let folds: std::collections::BTreeSet = pairs.iter().map(|p| p.2).collect(); + let mut accuracies = Vec::new(); + for fold in folds { + let train: Vec<(f64, bool)> = pairs.iter().filter(|p| p.2 != fold).filter_map(|p| Some((p.0?, p.1))).collect(); + let test: Vec<&(Option, bool, usize)> = pairs.iter().filter(|p| p.2 == fold).collect(); + let cutoff = best_cutoff(&train); + let right = test.iter().filter(|p| p.0.is_some_and(|score| (score >= cutoff) == p.1)).count(); + accuracies.push(right as f64 / test.len().max(1) as f64); + } + if accuracies.is_empty() { + return (0.0, 0.0); + } + let mean = accuracies.iter().sum::() / accuracies.len() as f64; + let variance = accuracies.iter().map(|a| (a - mean).powi(2)).sum::() / accuracies.len() as f64; + (mean, variance.sqrt()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn close(a: f64, b: f64) { + assert!((a - b).abs() < 1e-9, "{a} != {b}"); + } + + #[test] + fn the_best_cutoff_sits_between_the_scores_it_separates() { + // different 0.2, same 0.3, different 0.7, same 0.8: calling everything "same" gets 2 right, a cut-off in + // (0.2, 0.3] gets 3, one in (0.7, 0.8] also 3; the lowest of the best is the midpoint 0.25 + close(best_cutoff(&[(0.8, true), (0.2, false), (0.7, false), (0.3, true)]), 0.25); + // perfectly separable + close(best_cutoff(&[(0.1, false), (0.2, false), (0.8, true), (0.9, true)]), 0.5); + // every pair the same person: a cut-off below all of them + close(best_cutoff(&[(0.4, true), (0.6, true)]), -0.6); + // every pair a different person: above all of them + close(best_cutoff(&[(0.4, false), (0.6, false)]), 1.6); + // equal scores cannot be split: the pair stays together and the cut-off stays at the start + close(best_cutoff(&[(0.5, true), (0.5, false)]), -0.5); + // a non-finite score is ignored, and nothing at all has a neutral cut-off + close(best_cutoff(&[(f64::NAN, true), (0.1, false), (0.9, true)]), 0.5); + close(best_cutoff(&[]), 0.0); + } + + /// Three folds of two pairs. The cut-off for fold 0 is trained on folds 1 and 2 (different 0.2, same 0.3, different + /// 0.7, same 0.8: best 0.25), which gets fold 0's 0.9 same and 0.1 different both right: 1. For fold 1 it is trained + /// on 0.1 F, 0.3 T, 0.7 F, 0.9 T: best 0.2, which calls 0.8 right and 0.2 "same" wrongly: 1/2. For fold 2 it is + /// trained on 0.1 F, 0.2 F, 0.8 T, 0.9 T: best 0.5, which gets both of fold 2's inverted pairs wrong: 0. + /// Mean (1 + 1/2 + 0) / 3 = 1/2; variance (1/4 + 0 + 1/4) / 3 = 1/6. + #[test] + fn fold_accuracy_by_hand() { + let pairs = + [(Some(0.9), true, 0), (Some(0.1), false, 0), (Some(0.8), true, 1), (Some(0.2), false, 1), (Some(0.3), true, 2), (Some(0.7), false, 2)]; + let (mean, std) = fold_accuracy(&pairs); + close(mean, 0.5); + close(std, (1.0f64 / 6.0).sqrt()); + // separable in every fold: all right, no spread + let clean = [(Some(0.9), true, 0), (Some(0.1), false, 0), (Some(0.8), true, 1), (Some(0.2), false, 1)]; + assert_eq!(fold_accuracy(&clean), (1.0, 0.0)); + // no pairs: zero, not a division by zero + assert_eq!(fold_accuracy(&[]), (0.0, 0.0)); + } + + #[test] + fn a_pair_without_a_score_is_wrong_and_does_not_move_the_cut_off() { + // two separable folds; fold 0 also holds a different-person pair whose photo failed (no score) and fold 1 two same-person ones + let pairs = [ + (Some(0.9), true, 0), + (Some(0.1), false, 0), + (None, false, 0), + (Some(0.8), true, 1), + (Some(0.2), false, 1), + (None, true, 1), + (None, true, 1), + ]; + // fold 0: 2 of 3 right (the failed different pair gets no credit); fold 1: 2 of 4 right + let (mean, std) = fold_accuracy(&pairs); + close(mean, (2.0 / 3.0 + 0.5) / 2.0); + close(std, (2.0f64 / 3.0 - 0.5).abs() / 2.0); + // failures are not training data: without them every fold is separable + let scored: Vec<_> = pairs.iter().filter(|p| p.0.is_some()).copied().collect(); + close(fold_accuracy(&scored).0, 1.0); + // nothing scored at all: every pair is wrong + close(fold_accuracy(&[(None, true, 0), (None, false, 0), (None, true, 1), (None, false, 1)]).0, 0.0); + } + + #[test] + fn iou_counts_pixels_inclusively() { + let a = [0.0, 0.0, 9.0, 9.0]; // 10 x 10 pixels + close(iou(&a, &a), 1.0); + close(iou(&a, &[20.0, 20.0, 29.0, 29.0]), 0.0); + // 5 x 10 shared of 10 x 10 and 10 x 10: 50 / 150 + close(iou(&a, &[5.0, 0.0, 14.0, 9.0]), 1.0 / 3.0); + // touching at one pixel column still shares a pixel under the inclusive convention + close(iou(&a, &[9.0, 0.0, 18.0, 9.0]), 10.0 / 190.0); + // a degenerate pair has no overlap rather than a division by zero + close(iou(&[5.0, 5.0, 4.0, 4.0], &[5.0, 5.0, 4.0, 4.0]), 0.0); + } + + #[test] + fn a_detection_hits_from_half_overlap() { + let gt = [[0.0, 0.0, 10.0, 10.0]]; // pixels 0..=10 under the +1 convention: 11 x 11 + // 11 x 11 box against an 11 x 6 detection sharing its top rows: 66 / 121 > 0.5 hits + let hit = [[0.0, 0.0, 10.0, 5.0, 1.0]]; + close(wider_ap(&[Image { gt: >, keep: &[1], preds: &hit }]), 1.0); + // 11 x 5: 55 / 121 < 0.5 misses + let miss = [[0.0, 0.0, 10.0, 4.0, 1.0]]; + close(wider_ap(&[Image { gt: >, keep: &[1], preds: &miss }]), 0.0); + } + + /// Two faces, three detections: a hit (score .75), a false positive (.5) and a hit (.25). Scores normalise to 1, .5 + /// and 0. At thresholds .999 to .501 only the first detection counts (precision 1, recall 1/2, 499 times); at .500 to + /// .001 the first two (precision 1/2, recall 1/2, 500 times); at 0 all three (precision 2/3, recall 1). The + /// precision envelope is 1 up to recall 1/2 and 2/3 beyond it: AP = 1/2 * 1 + 1/2 * 2/3 = 5/6. + #[test] + fn average_precision_by_hand() { + let gt = [[0.0, 0.0, 10.0, 10.0], [100.0, 100.0, 10.0, 10.0]]; + let preds = [[0.0, 0.0, 10.0, 10.0, 0.75], [300.0, 300.0, 10.0, 10.0, 0.5], [100.0, 100.0, 10.0, 10.0, 0.25]]; + close(wider_ap(&[Image { gt: >, keep: &[1, 2], preds: &preds }]), 5.0 / 6.0); + // the same detections without the false positive: a perfect curve + let clean = [preds[0], preds[2]]; + close(wider_ap(&[Image { gt: >, keep: &[1, 2], preds: &clean }]), 1.0); + } + + #[test] + fn faces_outside_the_keep_list_are_neither_missed_nor_punished() { + let gt = [[0.0, 0.0, 10.0, 10.0], [100.0, 100.0, 10.0, 10.0]]; + // only the first face counts; a detection on the second must not count as a false positive + let both = [[0.0, 0.0, 10.0, 10.0, 0.9], [100.0, 100.0, 10.0, 10.0, 0.8]]; + close(wider_ap(&[Image { gt: >, keep: &[1], preds: &both }]), 1.0); + // and not finding it costs no recall + close(wider_ap(&[Image { gt: >, keep: &[1], preds: &both[..1] }]), 1.0); + } + + #[test] + fn nothing_to_find_or_nothing_found_scores_zero() { + let gt = [[0.0, 0.0, 10.0, 10.0]]; + close(wider_ap(&[]), 0.0); + close(wider_ap(&[Image { gt: >, keep: &[1], preds: &[] }]), 0.0); + close(wider_ap(&[Image { gt: >, keep: &[], preds: &[[0.0, 0.0, 10.0, 10.0, 0.9]] }]), 0.0); + // a non-finite score is dropped, not propagated + close(wider_ap(&[Image { gt: >, keep: &[1], preds: &[[0.0, 0.0, 10.0, 10.0, f64::NAN]] }]), 0.0); + } +} diff --git a/xtask/src/ai_bench/mod.rs b/xtask/src/ai_bench/mod.rs new file mode 100644 --- /dev/null +++ b/xtask/src/ai_bench/mod.rs @@ -0,0 +1,226 @@ +//! `cargo xtask ai-bench`: measure how well LightCraft's own models do on public benchmarks, so the accuracy figures in the +//! docs can be reproduced. Two benchmarks so far: face detection (YuNet) on the WIDER FACE validation set and face +//! recognition (SFace, AuraFace) on LFW. +//! +//! ```text +//! cargo xtask ai-bench fetch --accept-terms [--set SET] [--dir DIR] +//! cargo xtask ai-bench run face-detection --set wider-face-val [--dir DIR] [--model FILE] [--limit N] [--threads N] +//! cargo xtask ai-bench run face-recognition --set lfw [--dir DIR] [--model FILE] [--limit N] [--threads N] +//! ``` + +mod fetch; +mod lfw; +mod mat; +mod metrics; +mod wider; + +use std::path::{Path, PathBuf}; + +use lightcraft_faces::runtime::Embedder; +use lightcraft_faces::{known, yunet}; + +pub const USAGE: &str = "\ + ai-bench fetch --accept-terms [--set wider-face-val|lfw] [--dir DIR] + download the benchmark photos and ground truth (WIDER FACE validation: CC BY-NC-ND 4.0; LFW: research + use only; both are kept on your machine, never committed) and the models they need into DIR (default + corpus/ai-bench), checking pinned SHA-256 sums; without --set both sets + ai-bench run face-detection --set wider-face-val [--dir DIR] [--model FILE] [--limit N] [--threads N] + run YuNet on every validation photo and print the WIDER FACE average precision for the easy, + medium and hard settings and the wall time (--limit N: N photos spread evenly, for a quick look; + --model: another YuNet-architecture .onnx, default DIR/models/face_detection_yunet_2023mar.onnx) + ai-bench run face-recognition --set lfw [--dir DIR] [--model FILE] [--limit N] [--threads N] + detect, align and embed the photos of LFW's 6,000 pairs the way Photo > Detect/Recognize Faces does and + print the 10-fold verification accuracy and the wall time for SFace and AuraFace (--limit N: N pairs + spread evenly; --model: one recognition .onnx LightCraft knows by its SHA-256 instead of those two) +"; + +/// What a recognition model's makers publish for LFW (percent) and the page that says so. +const PUBLISHED_LFW: [(&str, f64, &str); 2] = [ + ( + "sface-2021dec", + 99.40, + "https://github.com/opencv/opencv_zoo/blob/25f423d0e04c31a17254620e58febd7386da523b/models/face_recognition_sface/README.md", + ), + ("auraface-v1", 99.65, "https://huggingface.co/fal/AuraFace-v1/blob/af6d057c9b0ec4071d4c49c80e3539258798b609/README.md"), +]; + +struct Args { + dir: PathBuf, + model: Option, + limit: Option, + threads: usize, + accept_terms: bool, + set: Option, +} + +fn parse(root: &Path, args: &[&str]) -> Result { + let mut a = Args { + dir: root.join("corpus").join("ai-bench"), + model: None, + limit: None, + threads: std::thread::available_parallelism().map_or(4, |n| n.get()), + accept_terms: false, + set: None, + }; + let mut it = args.iter(); + while let Some(&flag) = it.next() { + if flag == "--accept-terms" { + a.accept_terms = true; + continue; + } + let value = it.next().ok_or_else(|| format!("{flag} needs a value"))?; + let number = |what: &str| value.parse::().map_err(|_| format!("{what} must be a whole number, not {value}")); + match flag { + "--dir" => a.dir = PathBuf::from(value), + "--model" => a.model = Some(PathBuf::from(value)), + "--set" => a.set = Some((*value).to_string()), + "--limit" => match number("--limit")? { + 0 => return Err("--limit must be at least 1 (omit it to use every photo or pair)".to_string()), + k => a.limit = Some(k), + }, + "--threads" => a.threads = number("--threads")?, + _ => return Err(format!("unknown option `{flag}`\n\n{USAGE}")), + } + } + Ok(a) +} + +pub fn run(root: &Path, args: &[&str]) -> Result<(), String> { + match args { + ["fetch", rest @ ..] => { + let a = parse(root, rest)?; + let sets = match a.set.as_deref() { + None => vec![&fetch::WIDER, &fetch::LFW], + Some(id) if id == fetch::WIDER.id => vec![&fetch::WIDER], + Some(id) if id == fetch::LFW.id => vec![&fetch::LFW], + Some(other) => return Err(format!("no benchmark set `{other}` (wider-face-val or lfw)")), + }; + fetch::run(&a.dir, a.accept_terms, &sets) + } + ["run", "face-detection", rest @ ..] => { + let a = parse(root, rest)?; + match a.set.as_deref() { + Some(id) if id == fetch::WIDER.id => face_detection(&a), + other => Err(format!("face-detection needs --set {} (the only set so far), not {}", fetch::WIDER.id, other.unwrap_or("nothing"))), + } + } + ["run", "face-recognition", rest @ ..] => { + let a = parse(root, rest)?; + match a.set.as_deref() { + Some(id) if id == fetch::LFW.id => face_recognition(&a), + other => Err(format!("face-recognition needs --set {} (the only set so far), not {}", fetch::LFW.id, other.unwrap_or("nothing"))), + } + } + _ => Err(format!("usage:\n{USAGE}")), + } +} + +/// The YuNet detector from `DIR/models` (or `model`), and whether it is the pinned 2023mar file. +fn load_detector(a: &Args, model: Option<&Path>) -> Result<(yunet::Detector, PathBuf, bool), String> { + let file = known::download(known::YUNET_ID).map(|d| d.file_name).ok_or("no YuNet file name")?; + let model = model.map_or_else(|| a.dir.join("models").join(file), Path::to_path_buf); + let bytes = std::fs::read(&model).map_err(|e| format!("{}: {e} (run `cargo xtask ai-bench fetch` or pass --model)", model.display()))?; + let detector = yunet::Detector::new(&bytes).map_err(|e| format!("{}: {e}", model.display()))?; + let pinned = crate::corpus::file_sha256(&model).is_ok_and(|h| h == known::YUNET_SHA256); + Ok((detector, model, pinned)) +} + +fn face_detection(a: &Args) -> Result<(), String> { + let (detector, model, pinned) = load_detector(a, a.model.as_deref())?; + let side = detector.input_side(); + println!( + "YuNet {} ({}), input {side} px, {} thread(s)", + model.display(), + if pinned { "the pinned 2023mar model" } else { "NOT the pinned 2023mar model" }, + a.threads + ); + let r = wider::run(&a.dir.join(fetch::WIDER.id), &detector, a.limit, a.threads)?; + println!( + "WIDER FACE validation, {} photos{}", + r.photos, + if r.failed > 0 { format!(" ({} failed, scored as no detections)", r.failed) } else { String::new() } + ); + for (setting, ap) in wider::SETTINGS.iter().zip(r.ap) { + println!(" AP {setting:<6} {ap:.1}"); + } + println!(" wall time {:.1} s ({:.0} ms per photo)", r.seconds, r.seconds * 1000.0 / r.photos.max(1) as f64); + Ok(()) +} + +/// The recognition models to measure: `--model`'s file, else the downloaded SFace and AuraFace that are in `DIR/models`. +fn recognisers(a: &Args) -> Result, String> { + let known_model = |path: &Path| { + let sha = crate::corpus::file_sha256(path)?; + known::lookup(&sha) + .filter(|m| m.role == lightcraft_faces::Role::Embedder) + .map(|m| (path.to_path_buf(), m)) + .ok_or_else(|| format!("{}: not a recognition model LightCraft knows by its SHA-256 ({sha})", path.display())) + }; + if let Some(model) = &a.model { + return Ok(vec![known_model(model)?]); + } + let mut found = Vec::new(); + for (id, _, _) in PUBLISHED_LFW { + let file = known::download(id).map(|d| d.file_name).ok_or_else(|| format!("no download of {id}"))?; + let path = a.dir.join("models").join(file); + if path.is_file() { + found.push(known_model(&path)?); + } else { + println!("skipping {id}: {} is not there (run `cargo xtask ai-bench fetch --set lfw`)", path.display()); + } + } + if found.is_empty() { + return Err(format!("no recognition model in {} (run `cargo xtask ai-bench fetch --set lfw`)", a.dir.join("models").display())); + } + Ok(found) +} + +fn face_recognition(a: &Args) -> Result<(), String> { + let (detector, detector_file, pinned) = load_detector(a, None)?; + println!( + "device: CPU (LightCraft's pure-Rust engine, one thread per photo), {} worker thread(s)\nYuNet {} ({}), input {} px, default options", + a.threads, + detector_file.display(), + if pinned { "the pinned 2023mar model" } else { "NOT the pinned 2023mar model" }, + detector.input_side() + ); + for (path, manifest) in recognisers(a)? { + let embedder = Embedder::load(&path, &manifest).map_err(|e| format!("{}: {e}", path.display()))?; + let r = lfw::run(&a.dir.join(fetch::LFW.id), &detector, &embedder, a.limit, a.threads)?; + println!("{} ({}), LFW view 2, {} pairs of {} photos", manifest.name, path.display(), r.pairs, r.photos); + println!(" 10-fold accuracy {:.2} % (standard deviation {:.2})", r.accuracy.0, r.accuracy.1); + if let Some((_, published, page)) = PUBLISHED_LFW.iter().find(|(id, _, _)| *id == manifest.id) { + println!(" published: {published:.2} % ({page})"); + } + if r.fallbacks > 0 { + println!(" {} photos had no face found: the centre was cut out", r.fallbacks); + } + if r.failed > 0 || r.failed_pairs > 0 { + println!( + " {} photos could not be read or embedded: {} of the {} pairs have no score and count as wrong", + r.failed, r.failed_pairs, r.pairs + ); + } + println!(" wall time {:.1} s ({:.0} ms per photo)", r.seconds, r.seconds * 1000.0 / r.photos.max(1) as f64); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_limit_of_zero_is_refused() { + let e = parse(Path::new("."), &["--limit", "0"]).err().unwrap_or_default(); + assert!(e.contains("--limit") && e.contains("at least 1"), "{e:?}"); + assert!(parse(Path::new("."), &["--limit", "1"]).is_ok()); + } + + #[test] + fn every_archive_of_every_set_is_unpacked_through_a_completion_marker() { + // lfw.tgz and both WIDER zips are archives; pairs.txt is the only plain file + let archives: Vec<(&str, bool)> = [&fetch::WIDER, &fetch::LFW].iter().flat_map(|s| s.archives()).collect(); + assert_eq!(archives, vec![("WIDER_val.zip", true), ("eval_tools.zip", true), ("lfw.tgz", true), ("pairs.txt", false)]); + } +} diff --git a/xtask/src/ai_bench/wider.rs b/xtask/src/ai_bench/wider.rs new file mode 100644 --- /dev/null +++ b/xtask/src/ai_bench/wider.rs @@ -0,0 +1,343 @@ +//! WIDER FACE validation: read the ground truth, run a detector on every photo, score the three settings. + +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lightcraft_faces::yunet::{Detector, Options}; + +use super::mat; +use super::metrics::{self, Image}; + +/// Lowest detection score kept: the protocol wants nearly every detection, the curve then sweeps the threshold. +const MIN_SCORE: f32 = 0.02; +/// Most detections per photo (the densest WIDER crowds hold well over a thousand faces). +const MAX_FACES: usize = 2000; +pub const SETTINGS: [&str; 3] = ["easy", "medium", "hard"]; + +/// The validation ground truth: per photo, its path and face boxes `[x, y, w, h]`; per setting, the 1-based faces it counts. +pub struct GroundTruth { + pub images: Vec, + pub boxes: Vec>, + pub keep: [Vec>; 3], +} + +fn read_mat(path: &Path) -> Result, String> { + let bytes = std::fs::read(path).map_err(|e| format!("{}: {e} (run `cargo xtask ai-bench fetch`)", path.display()))?; + mat::parse(&bytes).map_err(|e| format!("{}: {e}", path.display())) +} + +/// Read the ground truth under `dir`: `eval_tools/ground_truth/*.mat` (the evaluation toolkit's) and the photos in +/// `WIDER_val/images//.jpg`. +pub fn load(dir: &Path) -> Result { + let gt = dir.join("eval_tools").join("ground_truth"); + let main = read_mat(>.join("wider_face_val.mat"))?; + let events = mat::var(&main, "event_list")?.cells().ok_or("event_list is not a cell array")?; + let files = mat::var(&main, "file_list")?.cells().ok_or("file_list is not a cell array")?; + let boxes = mat::var(&main, "face_bbx_list")?.cells().ok_or("face_bbx_list is not a cell array")?; + if files.len() != events.len() || boxes.len() != events.len() { + return Err(format!("event_list has {} events, file_list {} and face_bbx_list {}", events.len(), files.len(), boxes.len())); + } + // per event: its name, photo names and box lists, the photo counts checked before anything is paired up + let mut per_event = Vec::with_capacity(events.len()); + for (e, ((event, files), boxes)) in events.iter().zip(files).zip(boxes).enumerate() { + let event = event.text().ok_or_else(|| format!("event {e}: event_list entry is not text"))?; + let files = files.cells().ok_or_else(|| format!("event {e}: file_list entry is not a cell of cells"))?; + let boxes = boxes.cells().ok_or_else(|| format!("event {e}: face_bbx_list entry is not a cell of cells"))?; + if files.len() != boxes.len() { + return Err(format!("event {e}: file_list has {} photos, face_bbx_list {}", files.len(), boxes.len())); + } + per_event.push((event, files, boxes)); + } + let mut keeps: Vec>> = Vec::new(); + for setting in SETTINGS { + let vars = read_mat(>.join(format!("wider_{setting}_val.mat")))?; + let lists = mat::var(&vars, "gt_list")?.cells().ok_or("gt_list is not a cell array")?; + if lists.len() != per_event.len() { + return Err(format!("the {setting} setting lists {} events, the ground truth {}", lists.len(), per_event.len())); + } + let mut flat = Vec::new(); + for (e, (event, (_, files, _))) in lists.iter().zip(&per_event).enumerate() { + let imgs = event.cells().ok_or_else(|| format!("event {e}: the {setting} setting is not a cell of cells"))?; + if imgs.len() != files.len() { + return Err(format!("event {e}: the {setting} setting lists {} photos, the ground truth {}", imgs.len(), files.len())); + } + for (i, img) in imgs.iter().enumerate() { + flat.push(face_indices(img, setting, e, i)?); + } + } + keeps.push(flat); + } + let root = dir.join("WIDER_val").join("images"); + let mut out_images = Vec::new(); + let mut out_boxes = Vec::new(); + for (e, (event, files, boxes)) in per_event.iter().enumerate() { + for (file, face) in files.iter().zip(*boxes) { + let name = file.text().ok_or_else(|| format!("event {e}: file_list entry is not text"))?; + out_images.push(root.join(event).join(format!("{name}.jpg"))); + let rows = face.rows().ok_or_else(|| format!("event {e}: face_bbx_list entry is not a matrix"))?; + out_boxes.push(rows.iter().filter_map(|r| <[f64; 4]>::try_from(r.get(..4)?).ok()).collect()); + } + } + let [easy, medium, hard]: [Vec>; 3] = keeps.try_into().map_err(|_| "three settings expected")?; + Ok(GroundTruth { images: out_images, boxes: out_boxes, keep: [easy, medium, hard] }) +} + +/// The 1-based face numbers one photo's `gt_list` entry counts; a fractional, non-positive or absurd number is an error. +fn face_indices(entry: &mat::Value, setting: &str, event: usize, photo: usize) -> Result, String> { + entry + .numbers() + .unwrap_or(&[]) + .iter() + .map(|&v| { + if v.is_finite() && v.fract() == 0.0 && (1.0..=1e9).contains(&v) { + Ok(v as usize) + } else { + Err(format!("event {event}, photo {photo}: the {setting} setting has the face index {v}, expected a whole number from 1")) + } + }) + .collect() +} + +/// A photo as 8-bit RGB: `(pixels, width, height)`. +pub(super) fn load_rgb(path: &Path) -> Result<(Vec, usize, usize), String> { + use zune_core::bytestream::ZCursor; + use zune_core::colorspace::ColorSpace; + use zune_core::options::DecoderOptions; + let bytes = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?; + let options = DecoderOptions::default().jpeg_set_out_colorspace(ColorSpace::RGB); + let mut decoder = zune_jpeg::JpegDecoder::new_with_options(ZCursor::new(&bytes), options); + let pixels = decoder.decode().map_err(|e| format!("{}: {e:?}", path.display()))?; + let info = decoder.info().ok_or_else(|| format!("{}: no image information", path.display()))?; + let (w, h) = (usize::from(info.width), usize::from(info.height)); + match pixels.len().checked_div(w.saturating_mul(h)) { + Some(3) => Ok((pixels, w, h)), + Some(1) => Ok((pixels.iter().flat_map(|g| [*g; 3]).collect(), w, h)), + _ => Err(format!("{}: unexpected pixel layout", path.display())), + } +} + +/// The detections of one photo as `[x, y, w, h, score]` in pixels. +fn detect(detector: &Detector, path: &Path) -> Result, String> { + let (rgb, w, h) = load_rgb(path)?; + // the benchmark's own score, overlap and count limits; any other option keeps the detector's default + #[allow(clippy::needless_update, reason = "options the detector gains later keep their defaults here")] + let options = Options { score: MIN_SCORE, nms_iou: nms_iou(), max_faces: MAX_FACES, ..Options::default() }; + let faces = detector.detect(&rgb, w, h, &options).map_err(|e| format!("{}: {e}", path.display()))?; + let (w, h) = (w as f64, h as f64); + Ok(faces + .iter() + .map(|f| [f64::from(f.x0) * w, f64::from(f.y0) * h, f64::from(f.x1 - f.x0) * w, f64::from(f.y1 - f.y0) * h, f64::from(f.score)]) + .collect()) +} + +/// The suppression overlap LightCraft ships for YuNet. +fn nms_iou() -> f32 { + lightcraft_faces::known::yunet().thresholds.nms_iou.unwrap_or(0.4) +} + +pub struct Run { + pub photos: usize, + /// Photos that could not be read or searched (scored as having no detections). + pub failed: usize, + pub seconds: f64, + /// Average precision in percent for easy, medium and hard. + pub ap: [f64; 3], +} + +/// Detect on every photo (or `limit` of them, spread evenly) on `threads` threads and score the three settings. +pub fn run(dir: &Path, detector: &Detector, limit: Option, threads: usize) -> Result { + let truth = load(dir)?; + let n = truth.images.len(); + let chosen: Vec = match limit { + Some(k) if k > 0 && k < n => (0..k).map(|i| i * n / k).collect(), + _ => (0..n).collect(), + }; + let next = AtomicUsize::new(0); + let finished = AtomicUsize::new(0); + let started = Instant::now(); + let step = (chosen.len() / 20).max(1); + let joined: Vec, String>)>, String>> = std::thread::scope(|scope| { + let workers: Vec<_> = (0..threads.clamp(1, 64)) + .map(|_| { + scope.spawn(|| { + let mut done = Vec::new(); + loop { + let at = next.fetch_add(1, Ordering::Relaxed); + let Some(path) = chosen.get(at).and_then(|&i| truth.images.get(i)) else { break }; + done.push((at, detect(detector, path))); + let count = finished.fetch_add(1, Ordering::Relaxed) + 1; + if count.is_multiple_of(step) { + eprintln!(" {count}/{} ({:.1}/s)", chosen.len(), count as f64 / started.elapsed().as_secs_f64().max(1e-9)); + } + } + done + }) + }) + .collect(); + workers.into_iter().map(|w| w.join().map_err(|_| "a detection worker stopped unexpectedly".to_string())).collect() + }); + let seconds = started.elapsed().as_secs_f64(); + let mut preds: Vec> = vec![Vec::new(); chosen.len()]; + let mut failed = 0; + for worker in joined { + for (at, result) in worker? { + match (result, preds.get_mut(at)) { + (Ok(found), Some(slot)) => *slot = found, + (Err(e), _) => { + if failed < 5 { + eprintln!(" warning: {e}"); + } + failed += 1; + } + (Ok(_), None) => {} + } + } + } + let mut ap = [0.0; 3]; + for (s, score) in ap.iter_mut().enumerate() { + let images: Vec = chosen + .iter() + .zip(&preds) + .filter_map(|(&i, p)| Some(Image { gt: truth.boxes.get(i)?, keep: truth.keep.get(s)?.get(i)?, preds: p })) + .collect(); + *score = metrics::wider_ap(&images) * 100.0; + } + Ok(Run { photos: chosen.len(), failed, seconds, ap }) +} + +#[cfg(test)] +mod tests { + use super::super::mat::tests::{cell, chars, double, file}; + use super::*; + + /// A scratch directory that is removed afterwards. + struct Scratch(PathBuf); + + impl Scratch { + fn new(name: &str) -> Scratch { + let dir = std::env::temp_dir().join(format!("ai-bench-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + assert!(std::fs::create_dir_all(dir.join("eval_tools").join("ground_truth")).is_ok()); + Scratch(dir) + } + + fn gt(&self, name: &str, bytes: &[u8]) { + assert!(std::fs::write(self.0.join("eval_tools").join("ground_truth").join(name), bytes).is_ok()); + } + } + + impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } + } + + /// One event with one photo "a" holding two faces; the settings count the faces in `keeps`. + fn write_truth(s: &Scratch, keeps: [&[f64]; 3]) { + // the boxes [1 2 3 4] and [10 20 30 40], column-major + let boxes = double("", 2, 4, &[1.0, 10.0, 2.0, 20.0, 3.0, 30.0, 4.0, 40.0]); + let main = file(&[ + cell("event_list", &[chars("", "0--Parade")]), + cell("file_list", &[cell("", &[chars("", "a")])]), + cell("face_bbx_list", &[cell("", &[boxes])]), + ]); + s.gt("wider_face_val.mat", &main); + for (setting, keep) in SETTINGS.iter().zip(keeps) { + let list = file(&[cell("gt_list", &[cell("", &[double("", keep.len() as i32, 1, keep)])])]); + s.gt(&format!("wider_{setting}_val.mat"), &list); + } + } + + #[test] + fn ground_truth_reads_boxes_photos_and_the_faces_each_setting_counts() { + let s = Scratch::new("reads"); + write_truth(&s, [&[1.0], &[1.0, 2.0], &[2.0]]); + let truth = load(&s.0).unwrap(); + assert_eq!(truth.images, vec![s.0.join("WIDER_val").join("images").join("0--Parade").join("a.jpg")]); + assert_eq!(truth.boxes, vec![vec![[1.0, 2.0, 3.0, 4.0], [10.0, 20.0, 30.0, 40.0]]]); + assert_eq!(truth.keep, [vec![vec![1]], vec![vec![1, 2]], vec![vec![2]]]); + } + + #[test] + fn a_missing_annotation_file_is_an_error_that_says_how_to_fetch_it() { + let s = Scratch::new("missing"); + let e = load(&s.0).err().unwrap(); + assert!(e.contains("wider_face_val.mat") && e.contains("ai-bench fetch"), "{e}"); + // the main file alone is not enough: the settings' files are named too + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + assert!(std::fs::remove_file(s.0.join("eval_tools").join("ground_truth").join("wider_hard_val.mat")).is_ok()); + assert!(load(&s.0).err().unwrap().contains("wider_hard_val.mat")); + } + + #[test] + fn malformed_annotation_files_are_errors_not_panics() { + let s = Scratch::new("malformed"); + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + // garbage, a truncated file and a valid file without the variable the loader needs + s.gt("wider_face_val.mat", b"this is not a MAT-file"); + assert!(load(&s.0).is_err()); + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + let good = std::fs::read(s.0.join("eval_tools").join("ground_truth").join("wider_easy_val.mat")).unwrap(); + s.gt("wider_easy_val.mat", &good[..good.len() - 5]); + assert!(load(&s.0).is_err()); + s.gt("wider_easy_val.mat", &file(&[double("other", 1, 1, &[1.0])])); + assert!(load(&s.0).err().unwrap().contains("gt_list")); + } + + #[test] + fn fractional_zero_and_negative_face_indices_are_errors_not_truncated() { + for bad in [1.5, 0.0, -1.0, f64::NAN] { + let s = Scratch::new("index"); + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + let list = file(&[cell("gt_list", &[cell("", &[double("", 2, 1, &[1.0, bad])])])]); + s.gt("wider_medium_val.mat", &list); + let e = load(&s.0).err().unwrap_or_default(); + assert!(e.contains("medium") && e.contains("face index"), "{bad}: {e:?}"); + } + } + + #[test] + fn event_and_photo_counts_are_checked_and_errors_name_the_event() { + // top level: two events, one file list + let s = Scratch::new("top"); + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + let boxes = double("", 1, 4, &[1.0, 2.0, 3.0, 4.0]); + let main = file(&[ + cell("event_list", &[chars("", "0--A"), chars("", "1--B")]), + cell("file_list", &[cell("", &[chars("", "a")])]), + cell("face_bbx_list", &[cell("", std::slice::from_ref(&boxes)), cell("", std::slice::from_ref(&boxes))]), + ]); + s.gt("wider_face_val.mat", &main); + let e = load(&s.0).err().unwrap_or_default(); + assert!(e.contains("event_list") && e.contains("file_list"), "{e:?}"); + // event 1 has two files but one box list + let main = file(&[ + cell("event_list", &[chars("", "0--A"), chars("", "1--B")]), + cell("file_list", &[cell("", &[chars("", "a")]), cell("", &[chars("", "b"), chars("", "c")])]), + cell("face_bbx_list", &[cell("", std::slice::from_ref(&boxes)), cell("", &[boxes])]), + ]); + s.gt("wider_face_val.mat", &main); + let e = load(&s.0).err().unwrap_or_default(); + assert!(e.contains("event 1"), "{e:?}"); + } + + #[test] + fn a_setting_with_the_wrong_event_count_is_an_error() { + let s = Scratch::new("events"); + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + let item = cell("", &[double("", 1, 1, &[1.0])]); + s.gt("wider_hard_val.mat", &file(&[cell("gt_list", &[item.clone(), item])])); + let e = load(&s.0).err().unwrap_or_default(); + assert!(e.contains("hard") && e.contains("2 events"), "{e:?}"); + } + + #[test] + fn settings_that_disagree_with_the_ground_truth_on_the_photo_count_are_an_error() { + let s = Scratch::new("count"); + write_truth(&s, [&[1.0], &[1.0], &[1.0]]); + let two = file(&[cell("gt_list", &[cell("", &[double("", 1, 1, &[1.0]), double("", 1, 1, &[1.0])])])]); + s.gt("wider_medium_val.mat", &two); + assert!(load(&s.0).err().unwrap().contains("event 0")); + } +} diff --git a/xtask/src/corpus.rs b/xtask/src/corpus.rs --- a/xtask/src/corpus.rs +++ b/xtask/src/corpus.rs @@ -46,7 +46,8 @@ Tests that use a corpus skip cleanly when it is absent. )) } -fn file_sha256(path: &Path) -> Result { +/// SHA-256 of a file as lowercase hex; `ai_bench` reuses it to verify pinned downloads. +pub(crate) fn file_sha256(path: &Path) -> Result { use sha2::{Digest, Sha256}; let mut file = std::fs::File::open(path).map_err(|e| format!("open {}: {e}", path.display()))?; let mut hash = Sha256::new(); diff --git a/xtask/src/main.rs b/xtask/src/main.rs --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -3,6 +3,7 @@ //! Pure Rust (std + serde_json; flate2/brotli for the web bundle). External tools (`cargo`, `curl`, `tar`) are //! invoked through `std::process::Command`. +mod ai_bench; mod assets; mod bench; mod ci_lock; @@ -33,6 +34,8 @@ usage: cargo xtask commands: assets every image/icon/font/media file is attributed in assets/ATTRIBUTION.md; no Adobe assets + ai-bench fetch|run + reproduce the face-detection accuracy on WIDER FACE validation; see docs/faces.md bench [FILE] [--strict] [--threshold PCT] run the render benchmark, append to target/bench/history.jsonl, compare CPU time with the previous run (default input: corpus/raw/arw-sony-a7m3-compressed.arw) @@ -75,6 +78,7 @@ fn main() -> ExitCode { Some("ico") => ico::run(&rest), Some("version") => version::run(&root(), &rest), Some("layers") => cmd_layers(), + Some("ai-bench") => ai_bench::run(&root(), &rest), Some("assets") => assets::run(&root()), Some("bench") => bench::run(&root(), &rest), Some("parity") => parity::run(&root(), rest.contains(&"--write")),