collective
Browse / github.com/storytold/lightcraft / patch
xtask: ai-bench measures face recognition on LFW
xtask: ai-bench measures face recognition on LFW Stacked on #3110: its two commits come first here; review only the last commit, "xtask: ai-bench measures face recognition on LFW". **What it adds.** `cargo xtask ai-bench run face-recognition --set lfw`: the verification accuracy of the recognition models the faces crate already knows (SFace 2021dec and AuraFace v1) on LFW's standard 6,000 pairs, 10 folds, with the best cut-off on the other nine folds applied to each. `cargo xtask ai-bench fetch --set lfw` downloads the data and models with pinned SHA-256 sums. **Why.** The faces feature has recognition accuracy claims with no way to reprod
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- sm18lr88 (imported from pull request #3112, open)
- Base
662df66f1c08bdb1b1f1d0c1c638c4cef936d0b0- Size
- +1827 −1 in 11 files
- Review
- pending
- Shared
- 2026-10-28
- Id
04e68e2a6a838a1fc4d5edab
How it went
No other operator has reported using it yet.
Diff
raw · Cargo.lock docs/faces.md xtask/Cargo.toml xtask/src/ai_bench/fetch.rs xtask/src/ai_bench/lfw.rs xtask/src/ai_bench/mat.rs xtask/src/ai_bench/metrics.rs xtask/src/ai_bench/mod.rs xtask/src/ai_bench/wider.rs xtask/src/corpus.rs xtask/src/main.rs
@@ -7486,8 +7486,11 @@ version = "0.14.0" dependencies = [ "brotli", "flate2",+ "lightcraft-faces", "serde_json", "sha2",+ "zune-core",+ "zune-jpeg", ] [[package]]@@ -235,6 +235,67 @@ The detector's output matches OpenCV's own YuNet on a 45-photo public-domain tes mean box overlap 0.97), including marble busts and paintings, with no false boxes on the landscape and architecture photos in the set. +### And how well does it find faces?++`cargo xtask ai-bench` measures the detector on the **WIDER FACE validation set** (3,226 photos, 39,708 faces) and prints the+average precision (AP) for the benchmark's three settings: easy, medium and hard (hard includes the smallest faces). The photos are the publisher's, not ours: they are licensed CC BY-NC-ND 4.0, for non-commercial research+use only, so the command downloads them to your machine only when you pass `--accept-terms`, and they are never part of+this repository.++```text+cargo xtask ai-bench fetch --accept-terms --set wider-face-val # WIDER_val.zip, the evaluation toolkit's ground truth+ # and YuNet, into corpus/ai-bench (SHA-256 checked; needs curl and tar or unzip)+cargo xtask ai-bench run face-detection --set wider-face-val+```++`--dir DIR` puts the files elsewhere (the set goes in `DIR/wider-face-val`), `--model FILE` measures another+YuNet-architecture `.onnx`, and `--limit N` scores N photos spread evenly, for a quick look. Detection uses the same+code as **Photo ▸ Detect Faces** (a 640-pixel input), keeps every box scoring at least 0.02 as the protocol expects, and+scores by the benchmark's published procedure: intersection over union of at least 0.5, scores normalised over the whole+set, 1,000 thresholds. The scoring is our own implementation of that description, not the toolkit's MATLAB code.++Measured with the pinned YuNet 2023mar model on an Intel Core i9-13950HX (32 threads, CPU only): **87.8 easy, 83.7 medium,+59.8 hard** AP in 41 s for the 3,226 photos. The published figures for this model are 88.44 / 86.56 / 75.03. The gap is widest on hard, which includes the smallest+faces, and those are what a 640-pixel input resolves worst (the limit noted above).++### And how well does it recognise them?++`cargo xtask ai-bench run face-recognition --set lfw` measures the recognition models LightCraft knows by hash (SFace and+AuraFace) on **LFW** (Labeled Faces in the Wild), the standard face-verification benchmark: 6,000 pairs of photos in ten+folds, half of them the same person. The photos are the University of Massachusetts Amherst's+([vis-www.cs.umass.edu/lfw](http://vis-www.cs.umass.edu/lfw/), read through the+[Internet Archive's copy of the page](https://web.archive.org/web/20191205130041/http://vis-www.cs.umass.edu/lfw/)): news+photos of public figures, published with no licence but for research on face verification only, and the publisher warns+that no result on them shows an algorithm to be suitable for commercial use. Like WIDER FACE they go to your machine only+when you pass `--accept-terms` and are never part of this repository. The publisher's own host no longer resolves, so+`fetch` downloads the Internet Archive's copies of its `lfw.tgz` (173 MB; the md5 it prints, `a17d05bd...`, matches) and+`pairs.txt`, pinned by SHA-256.++```text+cargo xtask ai-bench fetch --accept-terms --set lfw # lfw.tgz, pairs.txt, YuNet, SFace and AuraFace (about 470 MB)+cargo xtask ai-bench run face-recognition --set lfw+```++Each photo goes through the code **Photo ▸ Detect Faces** and the recognition scan use: YuNet at its default options finds the+faces, the one nearest the centre (LFW photos are centred on the person) is aligned to the ArcFace template from the+detector's five landmarks, and the model's embedding of it is compared by cosine. A photo where YuNet finds no face has its+centre cut out instead, and the count is printed (none were needed). A pair scores the cosine of its two embeddings; for each+fold the best cut-off on the other nine is applied to it, and the result is the mean accuracy over the ten folds. There is+no mirrored-face (flip) test, as the app does none. `--model FILE` measures one recognition `.onnx` LightCraft knows by its+SHA-256, and `--limit N` scores N pairs spread evenly (at least 20, so each of the ten folds has a same-person and a+different-person pair). A pair with a photo that could not be read or embedded gets no score: it is left out when the+cut-off is chosen, counts as wrong in its fold, and is reported.++Measured on an Intel Core i9-13950HX (32 threads, CPU only: LightCraft's pure-Rust engine), all 6,000 pairs of 7,701 photos:++| Model | Accuracy (10 folds) | Published | Wall time |+| --- | --- | --- | --- |+| SFace 2021dec | **99.47 %** (std 0.44) | 99.40 % ([OpenCV Zoo](https://github.com/opencv/opencv_zoo/blob/25f423d0e04c31a17254620e58febd7386da523b/models/face_recognition_sface/README.md), measured with its [LFW evaluation](https://github.com/opencv/opencv_zoo/blob/25f423d0e04c31a17254620e58febd7386da523b/tools/eval/README.md)) | 148 s |+| AuraFace v1 | **99.40 %** (std 0.42) | 99.65 % ([model card](https://huggingface.co/fal/AuraFace-v1/blob/af6d057c9b0ec4071d4c49c80e3539258798b609/README.md)) | 395 s |++SFace matches its published figure. AuraFace is a quarter of a point (about 15 pairs) short of its card, which does not say+how it was aligned or whether it used a flip test; we have not tried to close that gap.+ ## Suggesting who is in a photo With **Recognise faces** switched on in Settings ▸ Faces and a recognition model chosen (**Use**; the switch cannot be@@ -10,6 +10,10 @@ workspace = true [dependencies] serde_json = { workspace = true }+# `cargo xtask ai-bench` (pure Rust): LightCraft's own face detector and JPEG decoder, both already in the lockfile+lightcraft-faces = { workspace = true }+zune-jpeg = { workspace = true }+zune-core = { workspace = true } # corpus download checksums sha2 = { workspace = true } # precompressed web bundle (`cargo xtask web`); pure Rustnew file mode 100644@@ -0,0 +1,192 @@+//! `cargo xtask ai-bench fetch`: download a benchmark set and the models it needs into the benchmark folder, check each+//! file against its pinned SHA-256 and unpack the archives. Needs `curl` and `tar` (or `unzip` for a zip), as+//! `cargo xtask corpus` needs `curl`.++use std::path::Path;+use std::process::Command;++use lightcraft_faces::known;++/// A file to download: name, URL, SHA-256 and whether it is an archive to unpack.+struct File {+ name: &'static str,+ url: &'static str,+ sha256: &'static str,+ archive: bool,+}++/// A benchmark set: its folder name, the terms the user accepts by passing `--accept-terms`, its files and the models+/// (by id in `lightcraft_faces::known`) its benchmark runs.+pub struct Set {+ pub id: &'static str,+ terms: &'static str,+ files: &'static [File],+ pub models: &'static [&'static str],+}++impl Set {+ /// Each file's name and whether it is an archive (unpacked, with a completion marker).+ #[cfg(test)]+ pub fn archives(&self) -> Vec<(&'static str, bool)> {+ self.files.iter().map(|f| (f.name, f.archive)).collect()+ }+}++pub const WIDER: Set = Set {+ id: "wider-face-val",+ terms: "WIDER FACE (http://shuoyang1213.me/WIDERFACE/) is licensed CC BY-NC-ND 4.0 (as its publisher's Hugging Face card states): \+non-commercial research use only, no derivatives. The photos are downloaded to your machine and used only to measure detection accuracy; they \+are never committed or redistributed.",+ files: &[+ File {+ name: "WIDER_val.zip",+ url: "https://huggingface.co/datasets/CUHK-CSE/wider_face/resolve/main/data/WIDER_val.zip",+ sha256: "f9efbd09f28c5d2d884be8c0eaef3967158c866a593fc36ab0413e4b2a58a17a",+ archive: true,+ },+ File {+ name: "eval_tools.zip",+ url: "http://shuoyang1213.me/WIDERFACE/support/eval_script/eval_tools.zip",+ sha256: "1cf49c8243fa8a1632efe7f6aa09fd7a3994ac838c9008bdbc9dd1d547bb234a",+ archive: true,+ },+ ],+ models: &[known::YUNET_ID],+};++/// The publisher's own host (vis-www.cs.umass.edu) no longer resolves, so the Internet Archive's copies of its+/// `lfw.tgz` and `pairs.txt` are used; the SHA-256 sums pin them, and `lfw.tgz` has the md5 the publisher printed.+pub const LFW: Set = Set {+ id: "lfw",+ terms: "LFW (Labeled Faces in the Wild, http://vis-www.cs.umass.edu/lfw/, University of Massachusetts Amherst) is a research benchmark \+of news photos of public figures. Its publisher gives no licence but warns that it is for research on face verification and \+must not be used to conclude that an algorithm is suitable for any commercial purpose. The photos are downloaded to your machine \+and used only to measure recognition accuracy; they are never committed or redistributed.",+ files: &[+ File {+ name: "lfw.tgz",+ url: "https://web.archive.org/web/20200310174752id_/http://vis-www.cs.umass.edu/lfw/lfw.tgz",+ sha256: "055f7d9c632d7370e6fb4afc7468d40f970c34a80d4c6f50ffec63f5a8d536c0",+ archive: true,+ },+ File {+ name: "pairs.txt",+ url: "https://web.archive.org/web/20191202153930id_/http://vis-www.cs.umass.edu/lfw/pairs.txt",+ sha256: "ea42330c62c92989f9d7c03237ed5d591365e89b3e649747777b70e692dc1592",+ archive: false,+ },+ ],+ models: &[known::YUNET_ID, "sface-2021dec", "auraface-v1"],+};++/// `curl` `url` to `dest`, unless `dest` already has `sha256`; a download with another hash is deleted and an error.+fn download(url: &str, dest: &Path, sha256: &str) -> Result<(), String> {+ if crate::corpus::file_sha256(dest).is_ok_and(|h| h == sha256) {+ return Ok(());+ }+ let mut curl = Command::new("curl");+ curl.args(["-fsSL", "--create-dirs", "-o"]).arg(dest).arg(url);+ crate::run(curl, &format!("curl {url}"))?;+ let actual = crate::corpus::file_sha256(dest)?;+ if actual != sha256 {+ std::fs::remove_file(dest).map_err(|e| format!("remove {}: {e}", dest.display()))?;+ return Err(format!("{}: sha256 is {actual}, expected {sha256} (file deleted; source {url})", dest.display()));+ }+ Ok(())+}++/// Unpack `archive` (a zip or a gzipped tar) into `into` with the system's bsdtar (Windows, macOS), else `unzip`.+fn unpack(archive: &Path, into: &Path) -> Result<(), String> {+ std::fs::create_dir_all(into).map_err(|e| format!("create {}: {e}", into.display()))?;+ let mut tar = Command::new("tar");+ tar.arg("-xf").arg(archive).arg("-C").arg(into);+ if crate::run(tar, &format!("tar -xf {}", archive.display())).is_ok() {+ return Ok(());+ }+ let mut unzip = Command::new("unzip");+ unzip.args(["-q", "-o"]).arg(archive).arg("-d").arg(into);+ crate::run(unzip, &format!("unzip {}", archive.display()))+}++/// Unpack `archive` (already checked against `sha256`) with `unpack` unless a completion marker for exactly that archive+/// exists. The marker is written, atomically, only after `unpack` succeeds: the unpacked folders can exist half-filled after an+/// interrupted extraction, so they never count as done and the next fetch extracts again over them.+fn unpack_once(set: &Path, archive: &str, sha256: &str, unpack: impl FnOnce() -> Result<(), String>) -> Result<(), String> {+ let marker = set.join(format!(".{archive}.unpacked"));+ if std::fs::read_to_string(&marker).is_ok_and(|done| done.trim() == sha256) {+ return Ok(());+ }+ unpack()?;+ let pending = set.join(format!(".{archive}.unpacked.tmp"));+ std::fs::write(&pending, sha256).map_err(|e| format!("write {}: {e}", pending.display()))?;+ std::fs::rename(&pending, &marker).map_err(|e| format!("rename {}: {e}", pending.display()))+}++/// Fetch everything the benchmarks of `sets` need under `dir`: each set's folder and `models/`.+pub fn run(dir: &Path, accept_terms: bool, sets: &[&Set]) -> Result<(), String> {+ if !accept_terms {+ let terms: Vec<&str> = sets.iter().map(|s| s.terms).collect();+ return Err(format!("{}\n\nPass --accept-terms to confirm you accept these terms.", terms.join("\n\n")));+ }+ for set in sets {+ let folder = dir.join(set.id);+ for file in set.files {+ download(file.url, &folder.join(file.name), file.sha256)?;+ if file.archive {+ unpack_once(&folder, file.name, file.sha256, || unpack(&folder.join(file.name), &folder))?;+ }+ }+ for id in set.models {+ let model = known::download(id).ok_or_else(|| format!("LightCraft offers no download of {id}"))?;+ download(&model.url, &dir.join("models").join(&model.file_name), &model.sha256)?;+ }+ }+ println!("fetched into {}", dir.display());+ Ok(())+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn an_interrupted_extraction_is_redone_and_a_finished_one_is_not() {+ let set = std::env::temp_dir().join(format!("ai-bench-fetch-{}", std::process::id()));+ let _ = std::fs::remove_dir_all(&set);+ assert!(std::fs::create_dir_all(&set).is_ok());+ // the extraction dies after creating the folder that used to count as "done"+ let interrupted = unpack_once(&set, "a.zip", "abc", || {+ std::fs::create_dir_all(set.join("WIDER_val").join("images")).map_err(|e| e.to_string())?;+ Err("interrupted".to_string())+ });+ assert!(interrupted.is_err());+ assert!(set.join("WIDER_val").join("images").is_dir());+ let mut runs = 0;+ assert!(+ unpack_once(&set, "a.zip", "abc", || {+ runs += 1;+ Ok(())+ })+ .is_ok()+ );+ assert_eq!(runs, 1, "the next fetch must extract again");+ assert!(+ unpack_once(&set, "a.zip", "abc", || {+ runs += 1;+ Ok(())+ })+ .is_ok()+ );+ assert_eq!(runs, 1, "a finished extraction is not repeated");+ // a different archive (hash) under the same name is extracted again+ assert!(+ unpack_once(&set, "a.zip", "def", || {+ runs += 1;+ Ok(())+ })+ .is_ok()+ );+ assert_eq!(runs, 2);+ let _ = std::fs::remove_dir_all(&set);+ }+}new file mode 100644@@ -0,0 +1,382 @@+//! LFW face verification ("view 2"): 6,000 pairs of photos in ten folds, half of them the same person. Every photo goes+//! through the code Photo ▸ Detect/Recognise Faces uses (the YuNet detector, alignment to the ArcFace template from its+//! landmarks, the installed recognition model) and a pair scores the cosine of its two embeddings.++use std::collections::HashMap;+use std::io::Read;+use std::path::{Path, PathBuf};+use std::sync::atomic::{AtomicUsize, Ordering};+use std::time::Instant;++use lightcraft_faces::align::{Rgb, align_to_template, crop_box};+use lightcraft_faces::matching::cosine;+use lightcraft_faces::runtime::Embedder;+use lightcraft_faces::yunet::{Detector, Options};++use super::metrics;+use super::wider::load_rgb;++/// How much wider than the guessed face box the fallback crop is (what the app does for a box without landmarks).+const BOX_PAD: f32 = 1.5;+/// Half the side of the fallback box, as a fraction of the shorter side of the photo (LFW centres the face).+const FALLBACK_HALF: f32 = 0.22;++/// Largest `pairs.txt` accepted (the real file is under 200 KB).+const MAX_PAIRS_BYTES: u64 = 1 << 20;++/// Refuse a `limit` that leaves some fold without a same-person and a different-person pair to evaluate and train on.+fn check_limit(limit: Option<usize>, pairs: &[Pair]) -> Result<(), String> {+ let folds = pairs.iter().map(|p| p.fold).max().map_or(0, |last| last.saturating_add(1));+ let minimum = folds.saturating_mul(2);+ match limit {+ Some(k) if k < minimum => Err(format!(+ "--limit {k} is too small to evaluate {folds} folds: use at least {minimum} pairs (a same-person and a different-person pair per fold)"+ )),+ _ => Ok(()),+ }+}++/// One line of `pairs.txt`.+#[derive(Debug, PartialEq)]+pub struct Pair {+ pub a: PathBuf,+ pub b: PathBuf,+ pub same: bool,+ pub fold: usize,+}++/// Read `dir/pairs.txt` (LFW's view 2) and check that every photo it names is under `dir/lfw`. The file starts with+/// `<folds> <pairs per half fold>`; each fold then lists that many same-person lines (`name n m`) followed by as+/// many different-person lines (`name n other m`).+pub fn load_pairs(dir: &Path) -> Result<Vec<Pair>, String> {+ let list = dir.join("pairs.txt");+ let unreadable = |e: std::io::Error| format!("{}: {e} (run `cargo xtask ai-bench fetch --set lfw`)", list.display());+ let file = std::fs::File::open(&list).map_err(unreadable)?;+ let size = file.metadata().map_err(unreadable)?.len();+ if size > MAX_PAIRS_BYTES {+ return Err(format!("{}: is {size} bytes, more than the {MAX_PAIRS_BYTES} a pairs file can be", list.display()));+ }+ // read one byte past the cap too, in case the file grows while it is read+ let mut text = String::new();+ file.take(MAX_PAIRS_BYTES + 1).read_to_string(&mut text).map_err(unreadable)?;+ if text.len() as u64 > MAX_PAIRS_BYTES {+ return Err(format!("{}: grew past {MAX_PAIRS_BYTES} bytes while being read", list.display()));+ }+ let photos = dir.join("lfw");+ let photo = |name: &str, number: &str| -> Result<PathBuf, String> {+ let number: usize = number.parse().map_err(|_| format!("{}: `{number}` is not a photo number", list.display()))?;+ if name.is_empty() || name.contains(['/', '\\', ':']) || name.contains("..") {+ return Err(format!("{}: `{name}` is not a person's folder name", list.display()));+ }+ Ok(photos.join(name).join(format!("{name}_{number:04}.jpg")))+ };+ let mut lines = text.lines();+ let header: Vec<usize> = lines.next().unwrap_or("").split_whitespace().map_while(|t| t.parse().ok()).collect();+ let [folds, half] = header.as_slice() else {+ return Err(format!("{}: the first line must be `<folds> <pairs per half fold>`", list.display()));+ };+ // the header is data from the file: a damaged one must not overflow+ let too_big = || format!("{}: the header `{folds} {half}` is too large", list.display());+ let per_fold = half.checked_mul(2).filter(|n| *n > 0).ok_or_else(too_big)?;+ let expected = folds.checked_mul(per_fold).ok_or_else(too_big)?;+ let mut pairs = Vec::new();+ for (i, line) in lines.filter(|l| !l.trim().is_empty()).enumerate() {+ let fold = i / per_fold;+ let words: Vec<&str> = line.split_whitespace().collect();+ let (a, b, same) = match words.as_slice() {+ [name, n, m] => (photo(name, n)?, photo(name, m)?, true),+ [name, n, other, m] => (photo(name, n)?, photo(other, m)?, false),+ _ => return Err(format!("{}: pair {} is malformed: `{line}`", list.display(), i + 1)),+ };+ if same != (i % per_fold < *half) {+ let (place, kind) = if same { ("different-person", "three fields") } else { ("same-person", "four fields") };+ return Err(format!("{}: pair {} is in the {place} half of fold {} but has {kind}: `{line}`", list.display(), i + 1, fold + 1));+ }+ pairs.push(Pair { a, b, same, fold });+ }+ if pairs.len() != expected {+ return Err(format!("{}: lists {} pairs, its header says {expected}", list.display(), pairs.len()));+ }+ if let Some(missing) = pairs.iter().flat_map(|p| [&p.a, &p.b]).find(|p| !p.is_file()) {+ return Err(format!("{}: no such photo (run `cargo xtask ai-bench fetch --set lfw`)", missing.display()));+ }+ Ok(pairs)+}++/// Run `work(0..n)` on `threads` threads and give the results in order.+fn parallel<T: Send>(n: usize, threads: usize, work: impl Fn(usize) -> T + Sync) -> Result<Vec<T>, String> {+ let next = AtomicUsize::new(0);+ let finished = AtomicUsize::new(0);+ let started = Instant::now();+ let step = (n / 20).max(1);+ let joined = std::thread::scope(|scope| {+ let workers: Vec<_> = (0..threads.clamp(1, 64))+ .map(|_| {+ scope.spawn(|| {+ let mut done = Vec::new();+ loop {+ let at = next.fetch_add(1, Ordering::Relaxed);+ if at >= n {+ break;+ }+ done.push((at, work(at)));+ let count = finished.fetch_add(1, Ordering::Relaxed) + 1;+ if count.is_multiple_of(step) {+ eprintln!(" {count}/{n} ({:.1}/s)", count as f64 / started.elapsed().as_secs_f64().max(1e-9));+ }+ }+ done+ })+ })+ .collect();+ workers.into_iter().map(|w| w.join().map_err(|_| "a worker stopped unexpectedly".to_string())).collect::<Result<Vec<_>, _>>()+ })?;+ let mut all: Vec<(usize, T)> = joined.into_iter().flatten().collect();+ all.sort_by_key(|(at, _)| *at);+ Ok(all.into_iter().map(|(_, value)| value).collect())+}++/// The face of one photo as the recogniser wants it, and whether the detector found none (so the centre was cut out).+/// LFW photos are centred on the person, so with several faces the one nearest the centre is the one.+fn face(detector: &Detector, path: &Path, size: (usize, usize)) -> Result<(Vec<u8>, bool), String> {+ let (rgb, w, h) = load_rgb(path)?;+ let faces = detector.detect(&rgb, w, h, &Options::default()).map_err(|e| format!("{}: {e}", path.display()))?;+ let picture = Rgb { data: &rgb, width: w, height: h };+ let (fw, fh) = (w as f32, h as f32);+ let distance = |f: &lightcraft_faces::yunet::Face| ((f.x0 + f.x1) / 2.0 - 0.5).hypot((f.y0 + f.y1) / 2.0 - 0.5);+ let aligned = faces+ .iter()+ .min_by(|a, b| distance(a).total_cmp(&distance(b)))+ .and_then(|f| align_to_template(&picture, &f.landmarks.map(|(x, y)| (x * fw, y * fh)), size.0, size.1));+ if let Some(crop) = aligned {+ return Ok((crop, false));+ }+ let half = fw.min(fh) * FALLBACK_HALF;+ let crop = crop_box(&picture, fw / 2.0 - half, fh / 2.0 - half, fw / 2.0 + half, fh / 2.0 + half, BOX_PAD, size.0, size.1)+ .ok_or_else(|| format!("{}: no face could be cut out", path.display()))?;+ Ok((crop, true))+}++/// A pair's score: the cosine of its two photos' embeddings; a pair with a photo that could not be embedded (or whose+/// embeddings cannot be compared) has none, so it is counted as wrong instead of being given a made-up score.+fn scores(pairs: &[Pair], embeddings: &HashMap<&Path, Vec<f32>>) -> Vec<(Option<f64>, bool, usize)> {+ pairs+ .iter()+ .map(|p| {+ let score = match (embeddings.get(p.a.as_path()), embeddings.get(p.b.as_path())) {+ (Some(a), Some(b)) => cosine(a, b).map(f64::from),+ _ => None,+ };+ (score, p.same, p.fold)+ })+ .collect()+}++pub struct Run {+ pub pairs: usize,+ pub photos: usize,+ /// Photos where the detector found no face and the centre was cut out.+ pub fallbacks: usize,+ /// Photos that could not be read or embedded.+ pub failed: usize,+ /// Pairs with such a photo: they have no score, take no part in choosing a cut-off and count as wrong.+ pub failed_pairs: usize,+ pub seconds: f64,+ /// Mean and standard deviation of the ten folds' accuracy, in percent.+ pub accuracy: (f64, f64),+}++/// Verify `embedder` on the pairs under `dir` (or `limit` of them, spread evenly) on `threads` threads.+pub fn run(dir: &Path, detector: &Detector, embedder: &Embedder, limit: Option<usize>, threads: usize) -> Result<Run, String> {+ let mut pairs = load_pairs(dir)?;+ check_limit(limit, &pairs)?;+ if let Some(k) = limit.filter(|k| *k > 0 && *k < pairs.len()) {+ let n = pairs.len();+ let keep: Vec<usize> = (0..k).map(|i| i * n / k).collect();+ pairs = pairs.into_iter().enumerate().filter(|(i, _)| keep.contains(i)).map(|(_, p)| p).collect();+ }+ let mut photos: Vec<&Path> = pairs.iter().flat_map(|p| [p.a.as_path(), p.b.as_path()]).collect();+ photos.sort();+ photos.dedup();+ let started = Instant::now();+ let size = embedder.input_size();+ let found = parallel(photos.len(), threads, |i| {+ let path = photos.get(i).ok_or("no such photo")?;+ let (crop, fallback) = face(detector, path, size)?;+ let embedding = embedder.embed(&crop).map_err(|e| format!("{}: {e}", path.display()))?;+ Ok::<_, String>((embedding, fallback))+ })?;+ let (mut embeddings, mut fallbacks, mut failed) = (HashMap::new(), 0, 0);+ for (path, result) in photos.iter().zip(found) {+ match result {+ Ok((embedding, fallback)) => {+ fallbacks += usize::from(fallback);+ embeddings.insert(*path, embedding);+ }+ Err(e) => {+ if failed < 5 {+ eprintln!(" warning: {e}");+ }+ failed += 1;+ }+ }+ }+ let scored = scores(&pairs, &embeddings);+ let failed_pairs = scored.iter().filter(|p| p.0.is_none()).count();+ let (mean, std) = metrics::fold_accuracy(&scored);+ Ok(Run {+ pairs: pairs.len(),+ failed_pairs,+ photos: photos.len(),+ fallbacks,+ failed,+ seconds: started.elapsed().as_secs_f64(),+ accuracy: (mean * 100.0, std * 100.0),+ })+}++#[cfg(test)]+mod tests {+ use super::*;++ /// A scratch directory that is removed afterwards.+ struct Scratch(PathBuf);++ impl Scratch {+ fn new(name: &str) -> Scratch {+ let dir = std::env::temp_dir().join(format!("ai-bench-lfw-{name}-{}", std::process::id()));+ let _ = std::fs::remove_dir_all(&dir);+ assert!(std::fs::create_dir_all(&dir).is_ok());+ Scratch(dir)+ }++ fn pairs(&self, text: &str) {+ assert!(std::fs::write(self.0.join("pairs.txt"), text).is_ok());+ }++ fn photo(&self, name: &str, number: usize) {+ let folder = self.0.join("lfw").join(name);+ assert!(std::fs::create_dir_all(&folder).is_ok());+ assert!(std::fs::write(folder.join(format!("{name}_{number:04}.jpg")), b"x").is_ok());+ }+ }++ impl Drop for Scratch {+ fn drop(&mut self) {+ let _ = std::fs::remove_dir_all(&self.0);+ }+ }++ /// Two folds of one same-person and one different-person pair each.+ const TWO_FOLDS: &str = "2 1\nAnn\t1\t2\nAnn\t1\tBob\t1\nBob\t1\t2\nBob\t2\tAnn\t2\n";++ fn two_folds() -> Scratch {+ let s = Scratch::new(&format!("two-{:?}", std::thread::current().id()).replace(['(', ')'], ""));+ for (name, n) in [("Ann", 1), ("Ann", 2), ("Bob", 1), ("Bob", 2)] {+ s.photo(name, n);+ }+ s.pairs(TWO_FOLDS);+ s+ }++ #[test]+ fn pairs_read_into_folds_of_same_then_different() {+ let s = two_folds();+ let pairs = load_pairs(&s.0).unwrap();+ let lfw = s.0.join("lfw");+ assert_eq!(pairs.len(), 4);+ assert_eq!(pairs[0], Pair { a: lfw.join("Ann/Ann_0001.jpg"), b: lfw.join("Ann/Ann_0002.jpg"), same: true, fold: 0 });+ assert_eq!(pairs[1], Pair { a: lfw.join("Ann/Ann_0001.jpg"), b: lfw.join("Bob/Bob_0001.jpg"), same: false, fold: 0 });+ assert_eq!((pairs[2].same, pairs[2].fold, pairs[3].same, pairs[3].fold), (true, 1, false, 1));+ }++ #[test]+ fn a_pairs_file_over_the_size_cap_is_refused_with_its_size() {+ let s = Scratch::new("big");+ let size = (MAX_PAIRS_BYTES + 1) as usize;+ s.pairs(&" ".repeat(size));+ let e = load_pairs(&s.0).err().unwrap_or_default();+ assert!(e.contains(&size.to_string()) && e.contains(&MAX_PAIRS_BYTES.to_string()), "{e:?}");+ }++ #[test]+ fn a_pair_of_the_wrong_kind_for_its_place_in_the_fold_is_an_error() {+ let s = two_folds();+ for (text, why) in [+ ("2 1\nAnn 1 Bob 1\nAnn 1 2\nBob 1 2\nBob 2 Ann 2\n", "different-person pair in the same-person half"),+ ("2 1\nAnn 1 2\nAnn 1 2\nBob 1 2\nBob 2 Ann 2\n", "same-person pair in the different-person half"),+ ("2 1\nAnn 1 2\nAnn 1 Bob 1\nBob 2 Ann 2\nBob 1 2\n", "halves swapped in the second fold"),+ ] {+ s.pairs(text);+ let e = load_pairs(&s.0).err().unwrap_or_default();+ assert!(e.contains("pair "), "{why}: {e:?}");+ }+ }++ #[test]+ fn a_limit_too_small_for_ten_fold_evaluation_is_refused_and_states_the_minimum() {+ let s = Scratch::new("limit");+ let pair = |fold| Pair { a: s.0.join("a"), b: s.0.join("b"), same: true, fold };+ let pairs: Vec<Pair> = (0..10).map(pair).collect();+ for k in [1, 5, 19] {+ let e = check_limit(Some(k), &pairs).err().unwrap_or_default();+ assert!(e.contains("20") && e.contains("10 folds"), "{k}: {e:?}");+ }+ assert!(check_limit(Some(20), &pairs).is_ok() && check_limit(None, &pairs).is_ok());+ }++ #[test]+ fn a_missing_pairs_file_or_photo_is_an_error_that_says_how_to_fetch_it() {+ let s = Scratch::new("missing");+ assert!(load_pairs(&s.0).err().unwrap().contains("ai-bench fetch"));+ s.pairs(TWO_FOLDS);+ s.photo("Ann", 1);+ let e = load_pairs(&s.0).err().unwrap();+ assert!(e.contains("Ann_0002.jpg") && e.contains("ai-bench fetch"), "{e}");+ }++ #[test]+ fn malformed_pairs_files_are_errors_not_panics() {+ let s = two_folds();+ for (text, why) in [+ ("", "empty"),+ ("garbage\n", "no header"),+ ("2\nAnn 1 2\n", "one-number header"),+ ("18446744073709551615 2\nAnn 1 2\n", "a header that overflows the pair count"),+ ("2 9223372036854775807\nAnn 1 2\n", "a header that overflows the fold size"),+ ("0 0\n", "empty folds"),+ ("2 1\nAnn 1 2\nAnn 1 Bob 1\nBob 1 2\n", "a pair short"),+ ("2 1\nAnn 1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\nAnn 1 2\n", "a pair too many"),+ ("2 1\nAnn 1\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a line of two words"),+ ("2 1\nAnn x 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a photo number that is not a number"),+ ("2 1\nAnn -1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a negative photo number"),+ ("2 1\n../Ann 1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a name that climbs out"),+ ("2 1\nC:Ann 1 2\nAnn 1 Bob 1\nBob 1 2\nBob 2 Ann 2\n", "a drive in the name"),+ ] {+ s.pairs(text);+ assert!(load_pairs(&s.0).is_err(), "{why}");+ }+ // a pairs file that is not text+ assert!(std::fs::write(s.0.join("pairs.txt"), [0xff, 0xfe, 0x00]).is_ok());+ assert!(load_pairs(&s.0).is_err());+ }++ #[test]+ fn a_pair_scores_the_cosine_of_its_embeddings_and_a_missing_one_scores_lowest() {+ let s = two_folds();+ let pairs = load_pairs(&s.0).unwrap();+ let lfw = s.0.join("lfw");+ let path = |n: &str| lfw.join(n.split('_').next().unwrap_or("")).join(format!("{n}.jpg"));+ let (a1, a2, b1) = (path("Ann_0001"), path("Ann_0002"), path("Bob_0001"));+ let mut embeddings: HashMap<&Path, Vec<f32>> = HashMap::new();+ embeddings.insert(a1.as_path(), vec![1.0, 0.0]);+ embeddings.insert(a2.as_path(), vec![0.6, 0.8]);+ embeddings.insert(b1.as_path(), vec![0.0, 1.0]);+ let scored = scores(&pairs, &embeddings);+ assert_eq!(scored.len(), 4);+ assert!((scored[0].0.unwrap_or(f64::NAN) - 0.6).abs() < 1e-6 && scored[0].1 && scored[0].2 == 0);+ assert!(scored[1].0.unwrap_or(f64::NAN).abs() < 1e-6 && !scored[1].1);+ // Bob_0002 has no embedding: no score at all, not a low one+ assert_eq!((scored[2].0, scored[3].0), (None, None));+ }+}new file mode 100644@@ -0,0 +1,315 @@+//! A reader for MATLAB level-5 MAT-files (the format of the WIDER FACE ground truth), written from MathWorks' published+//! file-format description: numeric arrays, char arrays and cell arrays, compressed or not, little-endian. Sizes are+//! checked against the data, nesting is bounded, and anything unexpected is an error.++use std::io::Read;++const MAX_DEPTH: usize = 16;+/// Largest decompressed variable.+const MAX_INFLATED: u64 = 1 << 30;+/// Largest single dimension of a matrix (every real variable is far smaller; a hostile one must not size an allocation).+const MAX_DIM: usize = 1 << 30;++#[derive(Clone, Debug, PartialEq)]+pub enum Value {+ /// A numeric matrix, column-major, as f64.+ Num {+ dims: Vec<usize>,+ data: Vec<f64>,+ },+ Char(String),+ Cell(Vec<Value>),+ /// Any other class (struct, sparse, objects): present but not read.+ Other,+}++impl Value {+ pub fn cells(&self) -> Option<&[Value]> {+ match self {+ Value::Cell(items) => Some(items),+ _ => None,+ }+ }++ pub fn text(&self) -> Option<&str> {+ match self {+ Value::Char(text) => Some(text),+ _ => None,+ }+ }++ /// Rows of a numeric matrix (stored column-major); an empty matrix is no rows.+ pub fn rows(&self) -> Option<Vec<Vec<f64>>> {+ let Value::Num { dims, data } = self else { return None };+ let (r, c) = (dims.first().copied().unwrap_or(0), dims.get(1).copied().unwrap_or(1));+ if r.checked_mul(c) != Some(data.len()) {+ return None;+ }+ if data.is_empty() {+ return Some(Vec::new());+ }+ Some((0..r).map(|i| (0..c).filter_map(|j| data.get(i + j * r).copied()).collect()).collect())+ }++ /// All numbers of a numeric matrix.+ pub fn numbers(&self) -> Option<&[f64]> {+ match self {+ Value::Num { data, .. } => Some(data),+ _ => None,+ }+ }+}++fn bad(what: &str) -> String {+ format!("malformed MAT-file: {what}")+}++struct Reader<'a> {+ b: &'a [u8],+ at: usize,+}++impl<'a> Reader<'a> {+ fn bytes(&self, len: usize) -> Option<&'a [u8]> {+ self.b.get(self.at..self.at.checked_add(len)?)+ }++ fn u32(&mut self) -> Result<u32, String> {+ let word = self.bytes(4).and_then(|s| <[u8; 4]>::try_from(s).ok()).ok_or_else(|| bad("truncated tag"))?;+ self.at += 4;+ Ok(u32::from_le_bytes(word))+ }++ fn done(&self) -> bool {+ self.at >= self.b.len()+ }++ /// The next data element: its type and bytes.+ fn element(&mut self) -> Result<(u32, &'a [u8]), String> {+ let first = self.u32()?;+ if first >> 16 != 0 {+ // small element: type and size in one word, data in the next four bytes+ let (ty, size) = (first & 0xffff, (first >> 16) as usize);+ let data = self.bytes(size.min(4)).ok_or_else(|| bad("truncated small element"))?;+ self.at += 4;+ return Ok((ty, data));+ }+ let size = self.u32()? as usize;+ let data = self.bytes(size).ok_or_else(|| bad("an element runs past the end of the file"))?;+ let end = self.at + size;+ // compressed elements are not padded; everything else is padded to 8 bytes+ self.at = if first == 15 { end } else { end.checked_next_multiple_of(8).ok_or_else(|| bad("size overflow"))? };+ Ok((first, data))+ }+}++/// The values of a numeric data element of MAT type `ty`.+fn numbers(ty: u32, d: &[u8]) -> Result<Vec<f64>, String> {+ fn le<const N: usize>(d: &[u8]) -> impl Iterator<Item = [u8; N]> + '_ {+ d.as_chunks::<N>().0.iter().copied()+ }+ Ok(match ty {+ 1 => d.iter().map(|b| f64::from(*b as i8)).collect(),+ 2 => d.iter().map(|b| f64::from(*b)).collect(),+ 3 => le::<2>(d).map(|c| f64::from(i16::from_le_bytes(c))).collect(),+ 4 => le::<2>(d).map(|c| f64::from(u16::from_le_bytes(c))).collect(),+ 5 => le::<4>(d).map(|c| f64::from(i32::from_le_bytes(c))).collect(),+ 6 => le::<4>(d).map(|c| f64::from(u32::from_le_bytes(c))).collect(),+ 7 => le::<4>(d).map(|c| f64::from(f32::from_le_bytes(c))).collect(),+ 9 => le::<8>(d).map(f64::from_le_bytes).collect(),+ 12 => le::<8>(d).map(|c| i64::from_le_bytes(c) as f64).collect(),+ 13 => le::<8>(d).map(|c| u64::from_le_bytes(c) as f64).collect(),+ _ => return Err(bad(&format!("numeric data of type {ty}"))),+ })+}++/// A miMATRIX element's body.+fn matrix(body: &[u8], depth: usize) -> Result<(String, Value), String> {+ if depth > MAX_DEPTH {+ return Err(bad("nested too deeply"));+ }+ if body.is_empty() {+ return Ok((String::new(), Value::Num { dims: vec![0, 0], data: Vec::new() }));+ }+ let mut r = Reader { b: body, at: 0 };+ let (_, flags) = r.element()?;+ let class = flags.first().copied().ok_or_else(|| bad("no array flags"))?;+ let complex = flags.get(1).is_some_and(|f| f & 0x08 != 0);+ let (dty, dims) = r.element()?;+ let dims: Vec<usize> = numbers(dty, dims)?.into_iter().map(|d| if d.is_finite() && d >= 0.0 { d as usize } else { 0 }).collect();+ if dims.iter().any(|d| *d > MAX_DIM) {+ return Err(bad("a dimension larger than any real variable"));+ }+ let count = dims.iter().try_fold(1usize, |n, d| n.checked_mul(*d)).ok_or_else(|| bad("dimensions overflow"))?;+ let (_, name) = r.element()?;+ let name = String::from_utf8_lossy(name).into_owned();+ let value = match class {+ 1 => {+ // every item takes at least a tag (8 bytes), which bounds a hostile count by the data+ if count > body.len() / 8 {+ return Err(bad("a cell array larger than its data"));+ }+ let mut items = Vec::with_capacity(count);+ for _ in 0..count {+ let (ty, b) = r.element()?;+ if ty != 14 {+ return Err(bad("a cell that is not a matrix"));+ }+ items.push(matrix(b, depth + 1)?.1);+ }+ Value::Cell(items)+ }+ 4 => {+ let (ty, d) = r.element()?;+ let text = match ty {+ 16 => String::from_utf8_lossy(d).into_owned(),+ _ => numbers(ty, d)?.into_iter().map(|c| char::from_u32(c as u32).unwrap_or('\u{fffd}')).collect(),+ };+ Value::Char(text)+ }+ 6..=15 => {+ let (ty, d) = r.element()?;+ let data = numbers(ty, d)?;+ if data.len() != count {+ return Err(bad("a matrix whose data does not match its dimensions"));+ }+ if complex && !r.done() {+ r.element()?;+ }+ Value::Num { dims, data }+ }+ _ => Value::Other,+ };+ Ok((name, value))+}++/// The variables in a MAT-file's bytes, by name.+pub fn parse(bytes: &[u8]) -> Result<Vec<(String, Value)>, String> {+ if bytes.len() < 128 {+ return Err(bad("shorter than its header"));+ }+ if bytes.get(126..128) != Some(b"IM") {+ return Err(bad("not a little-endian level-5 MAT-file (v7.3 HDF5 files are not supported)"));+ }+ let mut r = Reader { b: bytes, at: 128 };+ let mut out = Vec::new();+ while !r.done() {+ let (ty, body) = r.element()?;+ match ty {+ 14 => out.push(matrix(body, 0)?),+ 15 => {+ let mut inflated = Vec::new();+ flate2::read::ZlibDecoder::new(body)+ .take(MAX_INFLATED)+ .read_to_end(&mut inflated)+ .map_err(|e| bad(&format!("compressed variable: {e}")))?;+ let (ity, ibody) = Reader { b: &inflated, at: 0 }.element()?;+ if ity == 14 {+ out.push(matrix(ibody, 0)?);+ }+ }+ _ => {}+ }+ }+ Ok(out)+}++/// The variable `name` in `vars`.+pub fn var<'a>(vars: &'a [(String, Value)], name: &str) -> Result<&'a Value, String> {+ vars.iter().find(|(n, _)| n == name).map(|(_, v)| v).ok_or_else(|| format!("the MAT-file has no variable `{name}`"))+}++/// Builders for synthetic MAT-files, shared with the ground-truth tests.+#[cfg(test)]+pub(crate) mod tests {+ use super::*;++ fn element(ty: u32, data: &[u8]) -> Vec<u8> {+ let mut out = ty.to_le_bytes().to_vec();+ out.extend_from_slice(&(data.len() as u32).to_le_bytes());+ out.extend_from_slice(data);+ out.resize(out.len().div_ceil(8) * 8, 0);+ out+ }++ fn header(class: u8, dims: &[i32], name: &str) -> Vec<u8> {+ let mut out = element(6, &[class, 0, 0, 0, 0, 0, 0, 0]);+ out.extend(element(5, &dims.iter().flat_map(|d| d.to_le_bytes()).collect::<Vec<_>>()));+ out.extend(element(1, name.as_bytes()));+ out+ }++ /// A double matrix as a miMATRIX element (column-major `data`).+ pub(crate) fn double(name: &str, rows: i32, cols: i32, data: &[f64]) -> Vec<u8> {+ let mut body = header(6, &[rows, cols], name);+ body.extend(element(9, &data.iter().flat_map(|d| d.to_le_bytes()).collect::<Vec<_>>()));+ element(14, &body)+ }++ pub(crate) fn chars(name: &str, text: &str) -> Vec<u8> {+ let mut body = header(4, &[1, text.len() as i32], name);+ body.extend(element(16, text.as_bytes()));+ element(14, &body)+ }++ pub(crate) fn cell(name: &str, items: &[Vec<u8>]) -> Vec<u8> {+ let mut body = header(1, &[items.len() as i32, 1], name);+ for item in items {+ body.extend_from_slice(item);+ }+ element(14, &body)+ }++ /// A MAT-file: the 128-byte header, then `variables`.+ pub(crate) fn file(variables: &[Vec<u8>]) -> Vec<u8> {+ let mut out = vec![0u8; 128];+ out[126] = b'I';+ out[127] = b'M';+ for v in variables {+ out.extend_from_slice(v);+ }+ out+ }++ #[test]+ fn reads_numbers_text_and_nested_cells() {+ let bytes = file(&[double("m", 2, 2, &[1.0, 2.0, 3.0, 4.0]), cell("c", &[chars("", "ab"), cell("", &[double("", 1, 1, &[7.0])])])]);+ let vars = parse(&bytes).unwrap();+ // column-major [1 3; 2 4] has the rows [1, 3] and [2, 4]+ assert_eq!(var(&vars, "m").unwrap().rows(), Some(vec![vec![1.0, 3.0], vec![2.0, 4.0]]));+ let c = var(&vars, "c").unwrap().cells().unwrap();+ assert_eq!(c[0].text(), Some("ab"));+ assert_eq!(c[1].cells().unwrap()[0].numbers(), Some(&[7.0][..]));+ assert!(var(&vars, "absent").is_err());+ }++ #[test]+ fn a_matrix_without_data_has_no_rows_whatever_its_dimensions_claim() {+ // dims [N, 0] hold no data; building N empty rows from a hostile N would exhaust memory+ let empty = Value::Num { dims: vec![1 << 20, 0], data: Vec::new() };+ assert_eq!(empty.rows(), Some(Vec::new()));+ // a dimension no real variable has is refused at parse time+ let hostile = file(&[double("m", i32::MAX, 0, &[])]);+ assert!(parse(&hostile).is_err());+ // an ordinary empty matrix still parses and has no rows+ let vars = parse(&file(&[double("m", 0, 0, &[])])).unwrap();+ assert_eq!(var(&vars, "m").unwrap().rows(), Some(Vec::new()));+ }++ #[test]+ fn malformed_files_are_errors() {+ assert!(parse(b"too short").is_err());+ let mut wrong_endian = file(&[]);+ wrong_endian[126] = b'M';+ wrong_endian[127] = b'I';+ assert!(parse(&wrong_endian).is_err());+ let good = file(&[double("m", 1, 2, &[1.0, 2.0])]);+ for cut in [130, 140, good.len() - 1] {+ assert!(parse(&good[..cut]).is_err(), "cut at {cut}");+ }+ // a cell array that claims far more items than it has bytes+ let mut hostile = header(1, &[i32::MAX, 1], "c");+ hostile.extend(element(14, &[]));+ assert!(parse(&file(&[element(14, &hostile)])).is_err());+ }+}new file mode 100644@@ -0,0 +1,295 @@+//! WIDER FACE average precision, implemented from the published description of the evaluation (Yang et al., "WIDER+//! FACE", CVPR 2016, and the evaluation toolkit's documented procedure; no evaluation code was copied): detection+//! scores are normalised over the whole set, a detection hits a face when their intersection over union is at least 0.5+//! (with the toolkit's inclusive-pixel `+1` convention), faces outside the setting's keep list are neither counted nor+//! punished, precision and recall are taken at 1,000 score thresholds and the area under the curve is the VOC+//! all-point average precision.++/// How many score thresholds the precision/recall curve has.+const THRESHOLDS: usize = 1000;+/// Overlap at which a detection hits a face.+const MIN_IOU: f64 = 0.5;++/// One image: ground-truth boxes `[x, y, w, h]`, the 1-based indices of the faces this setting counts, and detections+/// `[x, y, w, h, score]`, all in pixels.+pub struct Image<'a> {+ pub gt: &'a [[f64; 4]],+ pub keep: &'a [usize],+ pub preds: &'a [[f64; 5]],+}++/// IoU of two `[x1, y1, x2, y2]` boxes, counting pixels inclusively (a box from 0 to 9 is ten pixels wide).+fn iou(a: &[f64; 4], b: &[f64; 4]) -> f64 {+ let iw = a[2].min(b[2]) - a[0].max(b[0]) + 1.0;+ let ih = a[3].min(b[3]) - a[1].max(b[1]) + 1.0;+ if iw <= 0.0 || ih <= 0.0 {+ return 0.0;+ }+ let area = |r: &[f64; 4]| (r[2] - r[0] + 1.0) * (r[3] - r[1] + 1.0);+ let inter = iw * ih;+ let union = area(a) + area(b) - inter;+ if union > 0.0 { inter / union } else { 0.0 }+}++/// VOC all-point average precision of a precision/recall curve (recall ascending).+fn voc_ap(recall: &[f64], precision: &[f64]) -> f64 {+ let mut mrec = vec![0.0];+ mrec.extend_from_slice(recall);+ mrec.push(1.0);+ let mut mpre = vec![0.0];+ mpre.extend_from_slice(precision);+ mpre.push(0.0);+ for i in (1..mpre.len()).rev() {+ mpre[i - 1] = mpre[i - 1].max(mpre[i]);+ }+ mrec.windows(2).zip(mpre.iter().skip(1)).filter(|(r, _)| r[1] != r[0]).map(|(r, p)| (r[1] - r[0]) * p).sum()+}++/// The WIDER FACE average precision (0 to 1) of one setting (easy, medium or hard: the `keep` lists) over `images`.+pub fn wider_ap(images: &[Image]) -> f64 {+ let (mut lo, mut hi) = (f64::INFINITY, f64::NEG_INFINITY);+ for p in images.iter().flat_map(|i| i.preds).filter(|p| p[4].is_finite()) {+ lo = lo.min(p[4]);+ hi = hi.max(p[4]);+ }+ let span = if hi > lo { hi - lo } else { 1.0 };+ // per threshold: detections kept that are not on an uncounted face, and counted faces found+ let mut curve = vec![(0.0f64, 0.0f64); THRESHOLDS];+ let mut faces = 0usize;+ for img in images {+ faces += img.keep.len();+ if img.gt.is_empty() || img.preds.is_empty() {+ continue;+ }+ let gt: Vec<[f64; 4]> = img.gt.iter().map(|g| [g[0], g[1], g[0] + g[2], g[1] + g[3]]).collect();+ let mut preds: Vec<([f64; 4], f64)> = img+ .preds+ .iter()+ .filter(|p| p[4].is_finite())+ .map(|p| ([p[0], p[1], p[0] + p[2], p[1] + p[3]], ((p[4] - lo) / span).clamp(0.0, 1.0)))+ .collect();+ preds.sort_by(|a, b| b.1.total_cmp(&a.1));+ // per face: whether this setting counts it, and 0 unseen / 1 found / -1 matched by a detection that is then ignored+ let mut state: Vec<(bool, i8)> = vec![(false, 0); gt.len()];+ for k in img.keep {+ if let Some(s) = k.checked_sub(1).and_then(|i| state.get_mut(i)) {+ s.0 = true;+ }+ }+ // after each detection (best first): the proposals so far and the counted faces found so far+ let (mut proposals, mut recalled) = (0usize, 0usize);+ let mut after: Vec<(usize, usize)> = Vec::with_capacity(preds.len());+ for (b, _) in &preds {+ let mut best = (0.0f64, None);+ for (j, g) in gt.iter().enumerate() {+ let o = iou(b, g);+ if o > best.0 {+ best = (o, Some(j));+ }+ }+ let mut ignored = false;+ if let (o, Some(j)) = best+ && o >= MIN_IOU+ && let Some((counted, found)) = state.get_mut(j)+ {+ if !*counted {+ if *found == 1 {+ recalled = recalled.saturating_sub(1);+ }+ *found = -1;+ ignored = true;+ } else if *found == 0 {+ *found = 1;+ recalled += 1;+ }+ }+ if !ignored {+ proposals += 1;+ }+ after.push((proposals, recalled));+ }+ for (t, point) in curve.iter_mut().enumerate() {+ let threshold = 1.0 - (t + 1) as f64 / THRESHOLDS as f64;+ if let Some((p, r)) = preds.iter().rposition(|p| p.1 >= threshold).and_then(|last| after.get(last)) {+ point.0 += *p as f64;+ point.1 += *r as f64;+ }+ }+ }+ if faces == 0 {+ return 0.0;+ }+ let precision: Vec<f64> = curve.iter().map(|c| if c.0 > 0.0 { c.1 / c.0 } else { 0.0 }).collect();+ let recall: Vec<f64> = curve.iter().map(|c| c.1 / faces as f64).collect();+ voc_ap(&recall, &precision)+}++/// The cut-off (a pair is "same" when its score is at least the cut-off) that gets most of `pairs` (score, same person)+/// right. Non-finite scores are ignored; among equally good cut-offs the lowest wins.+fn best_cutoff(pairs: &[(f64, bool)]) -> f64 {+ let mut sorted: Vec<(f64, bool)> = pairs.iter().copied().filter(|p| p.0.is_finite()).collect();+ sorted.sort_by(|a, b| a.0.total_cmp(&b.0));+ // a cut-off below every score calls every pair "same"+ let mut right = sorted.iter().filter(|p| p.1).count() as i64;+ let mut best = (right, sorted.first().map_or(0.0, |p| p.0 - 1.0));+ let mut groups = sorted.chunk_by(|a, b| a.0 == b.0).peekable();+ while let Some(group) = groups.next() {+ // raising the cut-off past this score turns its pairs into "different"+ right += group.iter().map(|p| if p.1 { -1 } else { 1 }).sum::<i64>();+ let score = group.first().map_or(0.0, |p| p.0);+ let cutoff = groups.peek().and_then(|next| next.first()).map_or(score + 1.0, |n| (score + n.0) / 2.0);+ if right > best.0 {+ best = (right, cutoff);+ }+ }+ best.1+}++/// LFW's verification accuracy: 10-fold cross-validation over `pairs` (score, same person, fold). For each fold the+/// cut-off that is best on the other folds is applied to it; the result is the mean and the standard deviation of the+/// folds' accuracies, as fractions. A pair without a score (`None`: a photo could not be embedded) takes no part in+/// choosing a cut-off and counts as wrong in its own fold.+pub fn fold_accuracy(pairs: &[(Option<f64>, bool, usize)]) -> (f64, f64) {+ let folds: std::collections::BTreeSet<usize> = pairs.iter().map(|p| p.2).collect();+ let mut accuracies = Vec::new();+ for fold in folds {+ let train: Vec<(f64, bool)> = pairs.iter().filter(|p| p.2 != fold).filter_map(|p| Some((p.0?, p.1))).collect();+ let test: Vec<&(Option<f64>, bool, usize)> = pairs.iter().filter(|p| p.2 == fold).collect();+ let cutoff = best_cutoff(&train);+ let right = test.iter().filter(|p| p.0.is_some_and(|score| (score >= cutoff) == p.1)).count();+ accuracies.push(right as f64 / test.len().max(1) as f64);+ }+ if accuracies.is_empty() {+ return (0.0, 0.0);+ }+ let mean = accuracies.iter().sum::<f64>() / accuracies.len() as f64;+ let variance = accuracies.iter().map(|a| (a - mean).powi(2)).sum::<f64>() / accuracies.len() as f64;+ (mean, variance.sqrt())+}++#[cfg(test)]+mod tests {+ use super::*;++ fn close(a: f64, b: f64) {+ assert!((a - b).abs() < 1e-9, "{a} != {b}");+ }++ #[test]+ fn the_best_cutoff_sits_between_the_scores_it_separates() {+ // different 0.2, same 0.3, different 0.7, same 0.8: calling everything "same" gets 2 right, a cut-off in+ // (0.2, 0.3] gets 3, one in (0.7, 0.8] also 3; the lowest of the best is the midpoint 0.25+ close(best_cutoff(&[(0.8, true), (0.2, false), (0.7, false), (0.3, true)]), 0.25);+ // perfectly separable+ close(best_cutoff(&[(0.1, false), (0.2, false), (0.8, true), (0.9, true)]), 0.5);+ // every pair the same person: a cut-off below all of them+ close(best_cutoff(&[(0.4, true), (0.6, true)]), -0.6);+ // every pair a different person: above all of them+ close(best_cutoff(&[(0.4, false), (0.6, false)]), 1.6);+ // equal scores cannot be split: the pair stays together and the cut-off stays at the start+ close(best_cutoff(&[(0.5, true), (0.5, false)]), -0.5);+ // a non-finite score is ignored, and nothing at all has a neutral cut-off+ close(best_cutoff(&[(f64::NAN, true), (0.1, false), (0.9, true)]), 0.5);+ close(best_cutoff(&[]), 0.0);+ }++ /// Three folds of two pairs. The cut-off for fold 0 is trained on folds 1 and 2 (different 0.2, same 0.3, different+ /// 0.7, same 0.8: best 0.25), which gets fold 0's 0.9 same and 0.1 different both right: 1. For fold 1 it is trained+ /// on 0.1 F, 0.3 T, 0.7 F, 0.9 T: best 0.2, which calls 0.8 right and 0.2 "same" wrongly: 1/2. For fold 2 it is+ /// trained on 0.1 F, 0.2 F, 0.8 T, 0.9 T: best 0.5, which gets both of fold 2's inverted pairs wrong: 0.+ /// Mean (1 + 1/2 + 0) / 3 = 1/2; variance (1/4 + 0 + 1/4) / 3 = 1/6.+ #[test]+ fn fold_accuracy_by_hand() {+ let pairs =+ [(Some(0.9), true, 0), (Some(0.1), false, 0), (Some(0.8), true, 1), (Some(0.2), false, 1), (Some(0.3), true, 2), (Some(0.7), false, 2)];+ let (mean, std) = fold_accuracy(&pairs);+ close(mean, 0.5);+ close(std, (1.0f64 / 6.0).sqrt());+ // separable in every fold: all right, no spread+ let clean = [(Some(0.9), true, 0), (Some(0.1), false, 0), (Some(0.8), true, 1), (Some(0.2), false, 1)];+ assert_eq!(fold_accuracy(&clean), (1.0, 0.0));+ // no pairs: zero, not a division by zero+ assert_eq!(fold_accuracy(&[]), (0.0, 0.0));+ }++ #[test]+ fn a_pair_without_a_score_is_wrong_and_does_not_move_the_cut_off() {+ // two separable folds; fold 0 also holds a different-person pair whose photo failed (no score) and fold 1 two same-person ones+ let pairs = [+ (Some(0.9), true, 0),+ (Some(0.1), false, 0),+ (None, false, 0),+ (Some(0.8), true, 1),+ (Some(0.2), false, 1),+ (None, true, 1),+ (None, true, 1),+ ];+ // fold 0: 2 of 3 right (the failed different pair gets no credit); fold 1: 2 of 4 right+ let (mean, std) = fold_accuracy(&pairs);+ close(mean, (2.0 / 3.0 + 0.5) / 2.0);+ close(std, (2.0f64 / 3.0 - 0.5).abs() / 2.0);+ // failures are not training data: without them every fold is separable+ let scored: Vec<_> = pairs.iter().filter(|p| p.0.is_some()).copied().collect();+ close(fold_accuracy(&scored).0, 1.0);+ // nothing scored at all: every pair is wrong+ close(fold_accuracy(&[(None, true, 0), (None, false, 0), (None, true, 1), (None, false, 1)]).0, 0.0);+ }++ #[test]+ fn iou_counts_pixels_inclusively() {+ let a = [0.0, 0.0, 9.0, 9.0]; // 10 x 10 pixels+ close(iou(&a, &a), 1.0);+ close(iou(&a, &[20.0, 20.0, 29.0, 29.0]), 0.0);+ // 5 x 10 shared of 10 x 10 and 10 x 10: 50 / 150+ close(iou(&a, &[5.0, 0.0, 14.0, 9.0]), 1.0 / 3.0);+ // touching at one pixel column still shares a pixel under the inclusive convention+ close(iou(&a, &[9.0, 0.0, 18.0, 9.0]), 10.0 / 190.0);+ // a degenerate pair has no overlap rather than a division by zero+ close(iou(&[5.0, 5.0, 4.0, 4.0], &[5.0, 5.0, 4.0, 4.0]), 0.0);+ }++ #[test]+ fn a_detection_hits_from_half_overlap() {+ let gt = [[0.0, 0.0, 10.0, 10.0]]; // pixels 0..=10 under the +1 convention: 11 x 11+ // 11 x 11 box against an 11 x 6 detection sharing its top rows: 66 / 121 > 0.5 hits+ let hit = [[0.0, 0.0, 10.0, 5.0, 1.0]];+ close(wider_ap(&[Image { gt: >, keep: &[1], preds: &hit }]), 1.0);+ // 11 x 5: 55 / 121 < 0.5 misses+ let miss = [[0.0, 0.0, 10.0, 4.0, 1.0]];+ close(wider_ap(&[Image { gt: >, keep: &[1], preds: &miss }]), 0.0);+ }++ /// Two faces, three detections: a hit (score .75), a false positive (.5) and a hit (.25). Scores normalise to 1, .5+ /// and 0. At thresholds .999 to .501 only the first detection counts (precision 1, recall 1/2, 499 times); at .500 to+ /// .001 the first two (precision 1/2, recall 1/2, 500 times); at 0 all three (precision 2/3, recall 1). The+ /// precision envelope is 1 up to recall 1/2 and 2/3 beyond it: AP = 1/2 * 1 + 1/2 * 2/3 = 5/6.+ #[test]+ fn average_precision_by_hand() {+ let gt = [[0.0, 0.0, 10.0, 10.0], [100.0, 100.0, 10.0, 10.0]];+ let preds = [[0.0, 0.0, 10.0, 10.0, 0.75], [300.0, 300.0, 10.0, 10.0, 0.5], [100.0, 100.0, 10.0, 10.0, 0.25]];+ close(wider_ap(&[Image { gt: >, keep: &[1, 2], preds: &preds }]), 5.0 / 6.0);+ // the same detections without the false positive: a perfect curve+ let clean = [preds[0], preds[2]];+ close(wider_ap(&[Image { gt: >, keep: &[1, 2], preds: &clean }]), 1.0);+ }++ #[test]+ fn faces_outside_the_keep_list_are_neither_missed_nor_punished() {+ let gt = [[0.0, 0.0, 10.0, 10.0], [100.0, 100.0, 10.0, 10.0]];+ // only the first face counts; a detection on the second must not count as a false positive+ let both = [[0.0, 0.0, 10.0, 10.0, 0.9], [100.0, 100.0, 10.0, 10.0, 0.8]];+ close(wider_ap(&[Image { gt: >, keep: &[1], preds: &both }]), 1.0);+ // and not finding it costs no recall+ close(wider_ap(&[Image { gt: >, keep: &[1], preds: &both[..1] }]), 1.0);+ }++ #[test]+ fn nothing_to_find_or_nothing_found_scores_zero() {+ let gt = [[0.0, 0.0, 10.0, 10.0]];+ close(wider_ap(&[]), 0.0);+ close(wider_ap(&[Image { gt: >, keep: &[1], preds: &[] }]), 0.0);+ close(wider_ap(&[Image { gt: >, keep: &[], preds: &[[0.0, 0.0, 10.0, 10.0, 0.9]] }]), 0.0);+ // a non-finite score is dropped, not propagated+ close(wider_ap(&[Image { gt: >, keep: &[1], preds: &[[0.0, 0.0, 10.0, 10.0, f64::NAN]] }]), 0.0);+ }+}new file mode 100644@@ -0,0 +1,226 @@+//! `cargo xtask ai-bench`: measure how well LightCraft's own models do on public benchmarks, so the accuracy figures in the+//! docs can be reproduced. Two benchmarks so far: face detection (YuNet) on the WIDER FACE validation set and face+//! recognition (SFace, AuraFace) on LFW.+//!+//! ```text+//! cargo xtask ai-bench fetch --accept-terms [--set SET] [--dir DIR]+//! cargo xtask ai-bench run face-detection --set wider-face-val [--dir DIR] [--model FILE] [--limit N] [--threads N]+//! cargo xtask ai-bench run face-recognition --set lfw [--dir DIR] [--model FILE] [--limit N] [--threads N]+//! ```++mod fetch;+mod lfw;+mod mat;+mod metrics;+mod wider;++use std::path::{Path, PathBuf};++use lightcraft_faces::runtime::Embedder;+use lightcraft_faces::{known, yunet};++pub const USAGE: &str = "\+ ai-bench fetch --accept-terms [--set wider-face-val|lfw] [--dir DIR]+ download the benchmark photos and ground truth (WIDER FACE validation: CC BY-NC-ND 4.0; LFW: research+ use only; both are kept on your machine, never committed) and the models they need into DIR (default+ corpus/ai-bench), checking pinned SHA-256 sums; without --set both sets+ ai-bench run face-detection --set wider-face-val [--dir DIR] [--model FILE] [--limit N] [--threads N]+ run YuNet on every validation photo and print the WIDER FACE average precision for the easy,+ medium and hard settings and the wall time (--limit N: N photos spread evenly, for a quick look;+ --model: another YuNet-architecture .onnx, default DIR/models/face_detection_yunet_2023mar.onnx)+ ai-bench run face-recognition --set lfw [--dir DIR] [--model FILE] [--limit N] [--threads N]+ detect, align and embed the photos of LFW's 6,000 pairs the way Photo > Detect/Recognize Faces does and+ print the 10-fold verification accuracy and the wall time for SFace and AuraFace (--limit N: N pairs+ spread evenly; --model: one recognition .onnx LightCraft knows by its SHA-256 instead of those two)+";++/// What a recognition model's makers publish for LFW (percent) and the page that says so.+const PUBLISHED_LFW: [(&str, f64, &str); 2] = [+ (+ "sface-2021dec",+ 99.40,+ "https://github.com/opencv/opencv_zoo/blob/25f423d0e04c31a17254620e58febd7386da523b/models/face_recognition_sface/README.md",+ ),+ ("auraface-v1", 99.65, "https://huggingface.co/fal/AuraFace-v1/blob/af6d057c9b0ec4071d4c49c80e3539258798b609/README.md"),+];++struct Args {+ dir: PathBuf,+ model: Option<PathBuf>,+ limit: Option<usize>,+ threads: usize,+ accept_terms: bool,+ set: Option<String>,+}++fn parse(root: &Path, args: &[&str]) -> Result<Args, String> {+ let mut a = Args {+ dir: root.join("corpus").join("ai-bench"),+ model: None,+ limit: None,+ threads: std::thread::available_parallelism().map_or(4, |n| n.get()),+ accept_terms: false,+ set: None,+ };+ let mut it = args.iter();+ while let Some(&flag) = it.next() {+ if flag == "--accept-terms" {+ a.accept_terms = true;+ continue;+ }+ let value = it.next().ok_or_else(|| format!("{flag} needs a value"))?;+ let number = |what: &str| value.parse::<usize>().map_err(|_| format!("{what} must be a whole number, not {value}"));+ match flag {+ "--dir" => a.dir = PathBuf::from(value),+ "--model" => a.model = Some(PathBuf::from(value)),+ "--set" => a.set = Some((*value).to_string()),+ "--limit" => match number("--limit")? {+ 0 => return Err("--limit must be at least 1 (omit it to use every photo or pair)".to_string()),+ k => a.limit = Some(k),+ },+ "--threads" => a.threads = number("--threads")?,+ _ => return Err(format!("unknown option `{flag}`\n\n{USAGE}")),+ }+ }+ Ok(a)+}++pub fn run(root: &Path, args: &[&str]) -> Result<(), String> {+ match args {+ ["fetch", rest @ ..] => {+ let a = parse(root, rest)?;+ let sets = match a.set.as_deref() {+ None => vec![&fetch::WIDER, &fetch::LFW],+ Some(id) if id == fetch::WIDER.id => vec![&fetch::WIDER],+ Some(id) if id == fetch::LFW.id => vec![&fetch::LFW],+ Some(other) => return Err(format!("no benchmark set `{other}` (wider-face-val or lfw)")),+ };+ fetch::run(&a.dir, a.accept_terms, &sets)+ }+ ["run", "face-detection", rest @ ..] => {+ let a = parse(root, rest)?;+ match a.set.as_deref() {+ Some(id) if id == fetch::WIDER.id => face_detection(&a),+ other => Err(format!("face-detection needs --set {} (the only set so far), not {}", fetch::WIDER.id, other.unwrap_or("nothing"))),+ }+ }+ ["run", "face-recognition", rest @ ..] => {+ let a = parse(root, rest)?;+ match a.set.as_deref() {+ Some(id) if id == fetch::LFW.id => face_recognition(&a),+ other => Err(format!("face-recognition needs --set {} (the only set so far), not {}", fetch::LFW.id, other.unwrap_or("nothing"))),+ }+ }+ _ => Err(format!("usage:\n{USAGE}")),+ }+}++/// The YuNet detector from `DIR/models` (or `model`), and whether it is the pinned 2023mar file.+fn load_detector(a: &Args, model: Option<&Path>) -> Result<(yunet::Detector, PathBuf, bool), String> {+ let file = known::download(known::YUNET_ID).map(|d| d.file_name).ok_or("no YuNet file name")?;+ let model = model.map_or_else(|| a.dir.join("models").join(file), Path::to_path_buf);+ let bytes = std::fs::read(&model).map_err(|e| format!("{}: {e} (run `cargo xtask ai-bench fetch` or pass --model)", model.display()))?;+ let detector = yunet::Detector::new(&bytes).map_err(|e| format!("{}: {e}", model.display()))?;+ let pinned = crate::corpus::file_sha256(&model).is_ok_and(|h| h == known::YUNET_SHA256);+ Ok((detector, model, pinned))+}++fn face_detection(a: &Args) -> Result<(), String> {+ let (detector, model, pinned) = load_detector(a, a.model.as_deref())?;+ let side = detector.input_side();+ println!(+ "YuNet {} ({}), input {side} px, {} thread(s)",+ model.display(),+ if pinned { "the pinned 2023mar model" } else { "NOT the pinned 2023mar model" },+ a.threads+ );+ let r = wider::run(&a.dir.join(fetch::WIDER.id), &detector, a.limit, a.threads)?;+ println!(+ "WIDER FACE validation, {} photos{}",+ r.photos,+ if r.failed > 0 { format!(" ({} failed, scored as no detections)", r.failed) } else { String::new() }+ );+ for (setting, ap) in wider::SETTINGS.iter().zip(r.ap) {+ println!(" AP {setting:<6} {ap:.1}");+ }+ println!(" wall time {:.1} s ({:.0} ms per photo)", r.seconds, r.seconds * 1000.0 / r.photos.max(1) as f64);+ Ok(())+}++/// The recognition models to measure: `--model`'s file, else the downloaded SFace and AuraFace that are in `DIR/models`.+fn recognisers(a: &Args) -> Result<Vec<(PathBuf, lightcraft_faces::ModelManifest)>, String> {+ let known_model = |path: &Path| {+ let sha = crate::corpus::file_sha256(path)?;+ known::lookup(&sha)+ .filter(|m| m.role == lightcraft_faces::Role::Embedder)+ .map(|m| (path.to_path_buf(), m))+ .ok_or_else(|| format!("{}: not a recognition model LightCraft knows by its SHA-256 ({sha})", path.display()))+ };+ if let Some(model) = &a.model {+ return Ok(vec![known_model(model)?]);+ }+ let mut found = Vec::new();+ for (id, _, _) in PUBLISHED_LFW {+ let file = known::download(id).map(|d| d.file_name).ok_or_else(|| format!("no download of {id}"))?;+ let path = a.dir.join("models").join(file);+ if path.is_file() {+ found.push(known_model(&path)?);+ } else {+ println!("skipping {id}: {} is not there (run `cargo xtask ai-bench fetch --set lfw`)", path.display());+ }+ }+ if found.is_empty() {+ return Err(format!("no recognition model in {} (run `cargo xtask ai-bench fetch --set lfw`)", a.dir.join("models").display()));+ }+ Ok(found)+}++fn face_recognition(a: &Args) -> Result<(), String> {+ let (detector, detector_file, pinned) = load_detector(a, None)?;+ println!(+ "device: CPU (LightCraft's pure-Rust engine, one thread per photo), {} worker thread(s)\nYuNet {} ({}), input {} px, default options",+ a.threads,+ detector_file.display(),+ if pinned { "the pinned 2023mar model" } else { "NOT the pinned 2023mar model" },+ detector.input_side()+ );+ for (path, manifest) in recognisers(a)? {+ let embedder = Embedder::load(&path, &manifest).map_err(|e| format!("{}: {e}", path.display()))?;+ let r = lfw::run(&a.dir.join(fetch::LFW.id), &detector, &embedder, a.limit, a.threads)?;+ println!("{} ({}), LFW view 2, {} pairs of {} photos", manifest.name, path.display(), r.pairs, r.photos);+ println!(" 10-fold accuracy {:.2} % (standard deviation {:.2})", r.accuracy.0, r.accuracy.1);+ if let Some((_, published, page)) = PUBLISHED_LFW.iter().find(|(id, _, _)| *id == manifest.id) {+ println!(" published: {published:.2} % ({page})");+ }+ if r.fallbacks > 0 {+ println!(" {} photos had no face found: the centre was cut out", r.fallbacks);+ }+ if r.failed > 0 || r.failed_pairs > 0 {+ println!(+ " {} photos could not be read or embedded: {} of the {} pairs have no score and count as wrong",+ r.failed, r.failed_pairs, r.pairs+ );+ }+ println!(" wall time {:.1} s ({:.0} ms per photo)", r.seconds, r.seconds * 1000.0 / r.photos.max(1) as f64);+ }+ Ok(())+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn a_limit_of_zero_is_refused() {+ let e = parse(Path::new("."), &["--limit", "0"]).err().unwrap_or_default();+ assert!(e.contains("--limit") && e.contains("at least 1"), "{e:?}");+ assert!(parse(Path::new("."), &["--limit", "1"]).is_ok());+ }++ #[test]+ fn every_archive_of_every_set_is_unpacked_through_a_completion_marker() {+ // lfw.tgz and both WIDER zips are archives; pairs.txt is the only plain file+ let archives: Vec<(&str, bool)> = [&fetch::WIDER, &fetch::LFW].iter().flat_map(|s| s.archives()).collect();+ assert_eq!(archives, vec![("WIDER_val.zip", true), ("eval_tools.zip", true), ("lfw.tgz", true), ("pairs.txt", false)]);+ }+}new file mode 100644@@ -0,0 +1,343 @@+//! WIDER FACE validation: read the ground truth, run a detector on every photo, score the three settings.++use std::path::{Path, PathBuf};+use std::sync::atomic::{AtomicUsize, Ordering};+use std::time::Instant;++use lightcraft_faces::yunet::{Detector, Options};++use super::mat;+use super::metrics::{self, Image};++/// Lowest detection score kept: the protocol wants nearly every detection, the curve then sweeps the threshold.+const MIN_SCORE: f32 = 0.02;+/// Most detections per photo (the densest WIDER crowds hold well over a thousand faces).+const MAX_FACES: usize = 2000;+pub const SETTINGS: [&str; 3] = ["easy", "medium", "hard"];++/// The validation ground truth: per photo, its path and face boxes `[x, y, w, h]`; per setting, the 1-based faces it counts.+pub struct GroundTruth {+ pub images: Vec<PathBuf>,+ pub boxes: Vec<Vec<[f64; 4]>>,+ pub keep: [Vec<Vec<usize>>; 3],+}++fn read_mat(path: &Path) -> Result<Vec<(String, mat::Value)>, String> {+ let bytes = std::fs::read(path).map_err(|e| format!("{}: {e} (run `cargo xtask ai-bench fetch`)", path.display()))?;+ mat::parse(&bytes).map_err(|e| format!("{}: {e}", path.display()))+}++/// Read the ground truth under `dir`: `eval_tools/ground_truth/*.mat` (the evaluation toolkit's) and the photos in+/// `WIDER_val/images/<event>/<name>.jpg`.+pub fn load(dir: &Path) -> Result<GroundTruth, String> {+ let gt = dir.join("eval_tools").join("ground_truth");+ let main = read_mat(>.join("wider_face_val.mat"))?;+ let events = mat::var(&main, "event_list")?.cells().ok_or("event_list is not a cell array")?;+ let files = mat::var(&main, "file_list")?.cells().ok_or("file_list is not a cell array")?;+ let boxes = mat::var(&main, "face_bbx_list")?.cells().ok_or("face_bbx_list is not a cell array")?;+ if files.len() != events.len() || boxes.len() != events.len() {+ return Err(format!("event_list has {} events, file_list {} and face_bbx_list {}", events.len(), files.len(), boxes.len()));+ }+ // per event: its name, photo names and box lists, the photo counts checked before anything is paired up+ let mut per_event = Vec::with_capacity(events.len());+ for (e, ((event, files), boxes)) in events.iter().zip(files).zip(boxes).enumerate() {+ let event = event.text().ok_or_else(|| format!("event {e}: event_list entry is not text"))?;+ let files = files.cells().ok_or_else(|| format!("event {e}: file_list entry is not a cell of cells"))?;+ let boxes = boxes.cells().ok_or_else(|| format!("event {e}: face_bbx_list entry is not a cell of cells"))?;+ if files.len() != boxes.len() {+ return Err(format!("event {e}: file_list has {} photos, face_bbx_list {}", files.len(), boxes.len()));+ }+ per_event.push((event, files, boxes));+ }+ let mut keeps: Vec<Vec<Vec<usize>>> = Vec::new();+ for setting in SETTINGS {+ let vars = read_mat(>.join(format!("wider_{setting}_val.mat")))?;+ let lists = mat::var(&vars, "gt_list")?.cells().ok_or("gt_list is not a cell array")?;+ if lists.len() != per_event.len() {+ return Err(format!("the {setting} setting lists {} events, the ground truth {}", lists.len(), per_event.len()));+ }+ let mut flat = Vec::new();+ for (e, (event, (_, files, _))) in lists.iter().zip(&per_event).enumerate() {+ let imgs = event.cells().ok_or_else(|| format!("event {e}: the {setting} setting is not a cell of cells"))?;+ if imgs.len() != files.len() {+ return Err(format!("event {e}: the {setting} setting lists {} photos, the ground truth {}", imgs.len(), files.len()));+ }+ for (i, img) in imgs.iter().enumerate() {+ flat.push(face_indices(img, setting, e, i)?);+ }+ }+ keeps.push(flat);+ }+ let root = dir.join("WIDER_val").join("images");+ let mut out_images = Vec::new();+ let mut out_boxes = Vec::new();+ for (e, (event, files, boxes)) in per_event.iter().enumerate() {+ for (file, face) in files.iter().zip(*boxes) {+ let name = file.text().ok_or_else(|| format!("event {e}: file_list entry is not text"))?;+ out_images.push(root.join(event).join(format!("{name}.jpg")));+ let rows = face.rows().ok_or_else(|| format!("event {e}: face_bbx_list entry is not a matrix"))?;+ out_boxes.push(rows.iter().filter_map(|r| <[f64; 4]>::try_from(r.get(..4)?).ok()).collect());+ }+ }+ let [easy, medium, hard]: [Vec<Vec<usize>>; 3] = keeps.try_into().map_err(|_| "three settings expected")?;+ Ok(GroundTruth { images: out_images, boxes: out_boxes, keep: [easy, medium, hard] })+}++/// The 1-based face numbers one photo's `gt_list` entry counts; a fractional, non-positive or absurd number is an error.+fn face_indices(entry: &mat::Value, setting: &str, event: usize, photo: usize) -> Result<Vec<usize>, String> {+ entry+ .numbers()+ .unwrap_or(&[])+ .iter()+ .map(|&v| {+ if v.is_finite() && v.fract() == 0.0 && (1.0..=1e9).contains(&v) {+ Ok(v as usize)+ } else {+ Err(format!("event {event}, photo {photo}: the {setting} setting has the face index {v}, expected a whole number from 1"))+ }+ })+ .collect()+}++/// A photo as 8-bit RGB: `(pixels, width, height)`.+pub(super) fn load_rgb(path: &Path) -> Result<(Vec<u8>, usize, usize), String> {+ use zune_core::bytestream::ZCursor;+ use zune_core::colorspace::ColorSpace;+ use zune_core::options::DecoderOptions;+ let bytes = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?;+ let options = DecoderOptions::default().jpeg_set_out_colorspace(ColorSpace::RGB);+ let mut decoder = zune_jpeg::JpegDecoder::new_with_options(ZCursor::new(&bytes), options);+ let pixels = decoder.decode().map_err(|e| format!("{}: {e:?}", path.display()))?;+ let info = decoder.info().ok_or_else(|| format!("{}: no image information", path.display()))?;+ let (w, h) = (usize::from(info.width), usize::from(info.height));+ match pixels.len().checked_div(w.saturating_mul(h)) {+ Some(3) => Ok((pixels, w, h)),+ Some(1) => Ok((pixels.iter().flat_map(|g| [*g; 3]).collect(), w, h)),+ _ => Err(format!("{}: unexpected pixel layout", path.display())),+ }+}++/// The detections of one photo as `[x, y, w, h, score]` in pixels.+fn detect(detector: &Detector, path: &Path) -> Result<Vec<[f64; 5]>, String> {+ let (rgb, w, h) = load_rgb(path)?;+ // the benchmark's own score, overlap and count limits; any other option keeps the detector's default+ #[allow(clippy::needless_update, reason = "options the detector gains later keep their defaults here")]+ let options = Options { score: MIN_SCORE, nms_iou: nms_iou(), max_faces: MAX_FACES, ..Options::default() };+ let faces = detector.detect(&rgb, w, h, &options).map_err(|e| format!("{}: {e}", path.display()))?;+ let (w, h) = (w as f64, h as f64);+ Ok(faces+ .iter()+ .map(|f| [f64::from(f.x0) * w, f64::from(f.y0) * h, f64::from(f.x1 - f.x0) * w, f64::from(f.y1 - f.y0) * h, f64::from(f.score)])+ .collect())+}++/// The suppression overlap LightCraft ships for YuNet.+fn nms_iou() -> f32 {+ lightcraft_faces::known::yunet().thresholds.nms_iou.unwrap_or(0.4)+}++pub struct Run {+ pub photos: usize,+ /// Photos that could not be read or searched (scored as having no detections).+ pub failed: usize,+ pub seconds: f64,+ /// Average precision in percent for easy, medium and hard.+ pub ap: [f64; 3],+}++/// Detect on every photo (or `limit` of them, spread evenly) on `threads` threads and score the three settings.+pub fn run(dir: &Path, detector: &Detector, limit: Option<usize>, threads: usize) -> Result<Run, String> {+ let truth = load(dir)?;+ let n = truth.images.len();+ let chosen: Vec<usize> = match limit {+ Some(k) if k > 0 && k < n => (0..k).map(|i| i * n / k).collect(),+ _ => (0..n).collect(),+ };+ let next = AtomicUsize::new(0);+ let finished = AtomicUsize::new(0);+ let started = Instant::now();+ let step = (chosen.len() / 20).max(1);+ let joined: Vec<Result<Vec<(usize, Result<Vec<[f64; 5]>, String>)>, String>> = std::thread::scope(|scope| {+ let workers: Vec<_> = (0..threads.clamp(1, 64))+ .map(|_| {+ scope.spawn(|| {+ let mut done = Vec::new();+ loop {+ let at = next.fetch_add(1, Ordering::Relaxed);+ let Some(path) = chosen.get(at).and_then(|&i| truth.images.get(i)) else { break };+ done.push((at, detect(detector, path)));+ let count = finished.fetch_add(1, Ordering::Relaxed) + 1;+ if count.is_multiple_of(step) {+ eprintln!(" {count}/{} ({:.1}/s)", chosen.len(), count as f64 / started.elapsed().as_secs_f64().max(1e-9));+ }+ }+ done+ })+ })+ .collect();+ workers.into_iter().map(|w| w.join().map_err(|_| "a detection worker stopped unexpectedly".to_string())).collect()+ });+ let seconds = started.elapsed().as_secs_f64();+ let mut preds: Vec<Vec<[f64; 5]>> = vec![Vec::new(); chosen.len()];+ let mut failed = 0;+ for worker in joined {+ for (at, result) in worker? {+ match (result, preds.get_mut(at)) {+ (Ok(found), Some(slot)) => *slot = found,+ (Err(e), _) => {+ if failed < 5 {+ eprintln!(" warning: {e}");+ }+ failed += 1;+ }+ (Ok(_), None) => {}+ }+ }+ }+ let mut ap = [0.0; 3];+ for (s, score) in ap.iter_mut().enumerate() {+ let images: Vec<Image> = chosen+ .iter()+ .zip(&preds)+ .filter_map(|(&i, p)| Some(Image { gt: truth.boxes.get(i)?, keep: truth.keep.get(s)?.get(i)?, preds: p }))+ .collect();+ *score = metrics::wider_ap(&images) * 100.0;+ }+ Ok(Run { photos: chosen.len(), failed, seconds, ap })+}++#[cfg(test)]+mod tests {+ use super::super::mat::tests::{cell, chars, double, file};+ use super::*;++ /// A scratch directory that is removed afterwards.+ struct Scratch(PathBuf);++ impl Scratch {+ fn new(name: &str) -> Scratch {+ let dir = std::env::temp_dir().join(format!("ai-bench-{name}-{}", std::process::id()));+ let _ = std::fs::remove_dir_all(&dir);+ assert!(std::fs::create_dir_all(dir.join("eval_tools").join("ground_truth")).is_ok());+ Scratch(dir)+ }++ fn gt(&self, name: &str, bytes: &[u8]) {+ assert!(std::fs::write(self.0.join("eval_tools").join("ground_truth").join(name), bytes).is_ok());+ }+ }++ impl Drop for Scratch {+ fn drop(&mut self) {+ let _ = std::fs::remove_dir_all(&self.0);+ }+ }++ /// One event with one photo "a" holding two faces; the settings count the faces in `keeps`.+ fn write_truth(s: &Scratch, keeps: [&[f64]; 3]) {+ // the boxes [1 2 3 4] and [10 20 30 40], column-major+ let boxes = double("", 2, 4, &[1.0, 10.0, 2.0, 20.0, 3.0, 30.0, 4.0, 40.0]);+ let main = file(&[+ cell("event_list", &[chars("", "0--Parade")]),+ cell("file_list", &[cell("", &[chars("", "a")])]),+ cell("face_bbx_list", &[cell("", &[boxes])]),+ ]);+ s.gt("wider_face_val.mat", &main);+ for (setting, keep) in SETTINGS.iter().zip(keeps) {+ let list = file(&[cell("gt_list", &[cell("", &[double("", keep.len() as i32, 1, keep)])])]);+ s.gt(&format!("wider_{setting}_val.mat"), &list);+ }+ }++ #[test]+ fn ground_truth_reads_boxes_photos_and_the_faces_each_setting_counts() {+ let s = Scratch::new("reads");+ write_truth(&s, [&[1.0], &[1.0, 2.0], &[2.0]]);+ let truth = load(&s.0).unwrap();+ assert_eq!(truth.images, vec![s.0.join("WIDER_val").join("images").join("0--Parade").join("a.jpg")]);+ assert_eq!(truth.boxes, vec![vec![[1.0, 2.0, 3.0, 4.0], [10.0, 20.0, 30.0, 40.0]]]);+ assert_eq!(truth.keep, [vec![vec![1]], vec![vec![1, 2]], vec![vec![2]]]);+ }++ #[test]+ fn a_missing_annotation_file_is_an_error_that_says_how_to_fetch_it() {+ let s = Scratch::new("missing");+ let e = load(&s.0).err().unwrap();+ assert!(e.contains("wider_face_val.mat") && e.contains("ai-bench fetch"), "{e}");+ // the main file alone is not enough: the settings' files are named too+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ assert!(std::fs::remove_file(s.0.join("eval_tools").join("ground_truth").join("wider_hard_val.mat")).is_ok());+ assert!(load(&s.0).err().unwrap().contains("wider_hard_val.mat"));+ }++ #[test]+ fn malformed_annotation_files_are_errors_not_panics() {+ let s = Scratch::new("malformed");+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ // garbage, a truncated file and a valid file without the variable the loader needs+ s.gt("wider_face_val.mat", b"this is not a MAT-file");+ assert!(load(&s.0).is_err());+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ let good = std::fs::read(s.0.join("eval_tools").join("ground_truth").join("wider_easy_val.mat")).unwrap();+ s.gt("wider_easy_val.mat", &good[..good.len() - 5]);+ assert!(load(&s.0).is_err());+ s.gt("wider_easy_val.mat", &file(&[double("other", 1, 1, &[1.0])]));+ assert!(load(&s.0).err().unwrap().contains("gt_list"));+ }++ #[test]+ fn fractional_zero_and_negative_face_indices_are_errors_not_truncated() {+ for bad in [1.5, 0.0, -1.0, f64::NAN] {+ let s = Scratch::new("index");+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ let list = file(&[cell("gt_list", &[cell("", &[double("", 2, 1, &[1.0, bad])])])]);+ s.gt("wider_medium_val.mat", &list);+ let e = load(&s.0).err().unwrap_or_default();+ assert!(e.contains("medium") && e.contains("face index"), "{bad}: {e:?}");+ }+ }++ #[test]+ fn event_and_photo_counts_are_checked_and_errors_name_the_event() {+ // top level: two events, one file list+ let s = Scratch::new("top");+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ let boxes = double("", 1, 4, &[1.0, 2.0, 3.0, 4.0]);+ let main = file(&[+ cell("event_list", &[chars("", "0--A"), chars("", "1--B")]),+ cell("file_list", &[cell("", &[chars("", "a")])]),+ cell("face_bbx_list", &[cell("", std::slice::from_ref(&boxes)), cell("", std::slice::from_ref(&boxes))]),+ ]);+ s.gt("wider_face_val.mat", &main);+ let e = load(&s.0).err().unwrap_or_default();+ assert!(e.contains("event_list") && e.contains("file_list"), "{e:?}");+ // event 1 has two files but one box list+ let main = file(&[+ cell("event_list", &[chars("", "0--A"), chars("", "1--B")]),+ cell("file_list", &[cell("", &[chars("", "a")]), cell("", &[chars("", "b"), chars("", "c")])]),+ cell("face_bbx_list", &[cell("", std::slice::from_ref(&boxes)), cell("", &[boxes])]),+ ]);+ s.gt("wider_face_val.mat", &main);+ let e = load(&s.0).err().unwrap_or_default();+ assert!(e.contains("event 1"), "{e:?}");+ }++ #[test]+ fn a_setting_with_the_wrong_event_count_is_an_error() {+ let s = Scratch::new("events");+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ let item = cell("", &[double("", 1, 1, &[1.0])]);+ s.gt("wider_hard_val.mat", &file(&[cell("gt_list", &[item.clone(), item])]));+ let e = load(&s.0).err().unwrap_or_default();+ assert!(e.contains("hard") && e.contains("2 events"), "{e:?}");+ }++ #[test]+ fn settings_that_disagree_with_the_ground_truth_on_the_photo_count_are_an_error() {+ let s = Scratch::new("count");+ write_truth(&s, [&[1.0], &[1.0], &[1.0]]);+ let two = file(&[cell("gt_list", &[cell("", &[double("", 1, 1, &[1.0]), double("", 1, 1, &[1.0])])])]);+ s.gt("wider_medium_val.mat", &two);+ assert!(load(&s.0).err().unwrap().contains("event 0"));+ }+}@@ -46,7 +46,8 @@ Tests that use a corpus skip cleanly when it is absent. )) } -fn file_sha256(path: &Path) -> Result<String, String> {+/// SHA-256 of a file as lowercase hex; `ai_bench` reuses it to verify pinned downloads.+pub(crate) fn file_sha256(path: &Path) -> Result<String, String> { use sha2::{Digest, Sha256}; let mut file = std::fs::File::open(path).map_err(|e| format!("open {}: {e}", path.display()))?; let mut hash = Sha256::new();@@ -3,6 +3,7 @@ //! Pure Rust (std + serde_json; flate2/brotli for the web bundle). External tools (`cargo`, `curl`, `tar`) are //! invoked through `std::process::Command`. +mod ai_bench; mod assets; mod bench; mod ci_lock;@@ -33,6 +34,8 @@ usage: cargo xtask <command> commands: assets every image/icon/font/media file is attributed in assets/ATTRIBUTION.md; no Adobe assets+ ai-bench fetch|run+ reproduce the face-detection accuracy on WIDER FACE validation; see docs/faces.md bench [FILE] [--strict] [--threshold PCT] run the render benchmark, append to target/bench/history.jsonl, compare CPU time with the previous run (default input: corpus/raw/arw-sony-a7m3-compressed.arw)@@ -75,6 +78,7 @@ fn main() -> ExitCode { Some("ico") => ico::run(&rest), Some("version") => version::run(&root(), &rest), Some("layers") => cmd_layers(),+ Some("ai-bench") => ai_bench::run(&root(), &rest), Some("assets") => assets::run(&root()), Some("bench") => bench::run(&root(), &rest), Some("parity") => parity::run(&root(), rest.contains(&"--write")),