collective
Browse / github.com/storytold/pdfcraft / patch
M1.9: deterministic save checks, SOURCE_DATE_EPOCH, UTF-8 text strings for 2.0 files (gap 22)
M1.9: deterministic save checks, SOURCE_DATE_EPOCH, UTF-8 text strings for 2.0 files (gap 22) ## What this changes Gap 22, three parts. I searched open and closed issues and PRs for deterministic save, UTF-8 strings and associated files and found nothing covering them. - **Deterministic save.** The `cos` writer already saved the same graph to the same bytes, but nothing checked it. A test now opens the same file several times (each open has its own hash state) and requires identical output for full and incremental saves, with and without object streams. The engine clock honours `SOURCE_DATE_EPOCH` (whole seconds; anything else is ignored; not on wasm), so the stamped `/ModDate` and e
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- Harrison-Booth (imported from pull request #4929, open)
- Base
9e90b2a54d6250465ba3d11a6eb0b13f3b690381- Size
- +160 −18 in 10 files
- Review
- pending
- Shared
- 2026-10-28
- Id
580ba48c2750f4dd72411665
How it went
No other operator has reported using it yet.
Diff
raw · ROADMAP.md crates/cos/src/document/objects.rs crates/cos/src/object.rs crates/cos/tests/it/writer.rs crates/engine/src/session/view.rs crates/organize/src/outline.rs crates/organize/src/tests.rs docs/gaps.md docs/pdf-spec-parity.md parity/acrobat-features.toml
@@ -200,6 +200,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md ## Progress log +- **2026-10-27 (M2, gap 22 partial):** Deterministic save is tested (separately opened documents save to identical bytes, full and incremental, with and without object streams) and `SOURCE_DATE_EPOCH` fixes the engine clock on non-wasm targets; `/AF` survives saves; `Document::text_string` writes UTF-8 for 2.0 files, used for bookmark titles. Other text writers, a UI switch, qpdf and veraPDF checks remain. Estimates unchanged. - **2026-10-27 (M10, OCR models):** Model staging downloads and verifies the ONNX pair after the upstream RTEN URL brownout; runtime discovery prefers complete ONNX pairs per directory, retains complete RTEN fallback and preserves directory precedence. - **2026-10-24 (M10, #4158 partial):** Word, HTML, RTF and PowerPoint export keep a block that continues another on its line (an italic case name, a link, a superscript) as a run in its own style inside that paragraph, with the source's word space, instead of a paragraph of its own. Style changes that wrap onto the next line still split paragraphs (the reading blocks keep style boundaries). Estimates unchanged. - **2026-10-24 (M3, window move on Wayland, #3970 follow-up):** After a move or resize by PdfCraft's own title bar or edges, the button release given back to egui goes before the pointer leaving the window, so hover no longer sticks at the old place until the pointer moves. Found by CodeRabbit in review of #3970; a unit test covers it. Estimates unchanged.@@ -2,6 +2,7 @@ use super::reader::find; use super::state::{LENGTHS, LoadGuard, ObjStm, ObjectReader, ReadState, Slot, SourceIdentity, objstm_member, owned_bytes}; use super::xref::{Revision, XrefEntry}; use super::*;+use crate::PdfString; impl Document { pub fn bytes(&self) -> &Arc<Vec<u8>> {@@ -29,6 +30,11 @@ impl Document { self.version = version.into(); } + /// A text string for this document: UTF-8 when its header says PDF 2.0, UTF-16BE before.+ pub fn text_string(&self, s: &str) -> PdfString {+ if self.version.starts_with("2.") { PdfString::text_utf8(s) } else { PdfString::text(s) }+ }+ pub fn trailer(&self) -> &Dict { &self.trailer }@@ -50,6 +50,18 @@ impl PdfString { Self { bytes, hex: false } } + /// A text string as PDF 2.0 writes it (ISO 32000-2 §7.9.2.2): PDFDocEncoding when possible,+ /// else UTF-8 with a BOM. Only for a document that is PDF 2.0; earlier readers do not know it.+ pub fn text_utf8(s: &str) -> Self {+ let plain = Self::text(s);+ if plain.bytes.first() != Some(&0xFE) {+ return plain;+ }+ let mut bytes = vec![0xEF, 0xBB, 0xBF];+ bytes.extend_from_slice(s.as_bytes());+ Self { bytes, hex: false }+ }+ /// Decode a text string: UTF-16BE (BOM), UTF-8 (BOM, PDF 2.0) or PDFDocEncoding. /// /// Lenient beyond the spec, as readers are: invalid sequences after a BOM decode lossily@@ -544,6 +556,15 @@ mod tests { assert!(!PdfString::text("abc").bytes.starts_with(&[0xFE, 0xFF])); } + #[test]+ fn utf8_text_strings_round_trip() {+ let s = PdfString::text_utf8("Caf\u{e9} \u{4e2d}\u{1f600}");+ assert_eq!(&s.bytes[..3], &[0xEF, 0xBB, 0xBF]);+ assert_eq!(s.to_text(), "Caf\u{e9} \u{4e2d}\u{1f600}");+ // Plain text needs no BOM in either form.+ assert_eq!(PdfString::text_utf8("abc"), PdfString::text("abc"));+ }+ #[test] fn text_string_encodings() { // UTF-16BE with BOM.@@ -513,3 +513,75 @@ fn incremental_saves_of_a_file_whose_offsets_count_from_its_first_byte_keep_coun // No qpdf check here: qpdf counts every offset from the header in a file with bytes before // it, so it already reconstructs this input before anything is appended. }++/// A 2.0 file with an associated file on the catalog, the page and a hidden stream.+fn af_fixture() -> Vec<u8> {+ Pdf::new()+ .version("2.0")+ .obj("<< /Type /Catalog /Pages 2 0 R /AF [5 0 R] >>")+ .obj("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 300] >>")+ .obj("<< /Type /Page /Parent 2 0 R /AF [5 0 R] >>")+ .obj("<< /Producer (x) >>")+ .obj("<< /Type /Filespec /F (a.xml) /UF (a.xml) /AFRelationship /Data /EF << /F 6 0 R >> >>")+ .stream("/Type /EmbeddedFile /Subtype /text#2Fxml", "<a/>")+ .trailer_entry("/Info 4 0 R")+ .build()+}++/// Whether `holder`'s `/AF` array still holds the fixture's file spec: relationship `Data` and+/// an embedded `<a/>` stream of type text/xml.+fn has_af(doc: &Document, holder: &Object) -> bool {+ let Some(spec) = holder+ .as_dict()+ .and_then(|d| d.get(b"AF"))+ .map(|o| doc.resolve(o))+ .and_then(|a| a.as_array().and_then(|a| a.first().map(|o| doc.resolve(o))))+ else {+ return false;+ };+ let Some(spec) = spec.as_dict() else { return false };+ let relationship = spec.get(b"AFRelationship").map(|o| doc.resolve(o));+ let file = spec.get(b"EF").map(|o| doc.resolve(o)).and_then(|ef| ef.as_dict().and_then(|d| d.get(b"F")).map(|o| doc.resolve(o)));+ matches!(relationship.as_deref().and_then(Object::as_name), Some(b"Data"))+ && matches!(file.as_deref(), Some(Object::Stream(s)) if s.dict.get(b"Subtype").map(|o| doc.resolve(o)).and_then(|o| o.as_name().map(<[u8]>::to_vec)).as_deref() == Some(b"text/xml".as_slice()))+}++fn af_survives(bytes: Vec<u8>) -> bool {+ let doc = Document::open(Arc::new(bytes)).unwrap();+ let root = doc.resolve(doc.trailer().get(b"Root").unwrap());+ let pages = root.as_dict().and_then(|d| d.get(b"Pages")).map(|o| doc.resolve(o));+ let page = pages+ .as_deref()+ .and_then(Object::as_dict)+ .and_then(|d| d.get(b"Kids"))+ .map(|o| doc.resolve(o))+ .and_then(|k| k.as_array().and_then(|k| k.first().map(|o| doc.resolve(o))));+ has_af(&doc, &root) && page.is_some_and(|p| has_af(&doc, &p))+}++#[test]+fn saves_are_byte_identical_across_runs_and_keep_associated_files() {+ for bytes in [af_fixture(), fixture()] {+ for object_streams in [true, false] {+ let opts = SaveOptions { object_streams, mod_date: Some("D:20260101000000Z".into()), ..SaveOptions::default() };+ // Each open builds its own hash state, so an order that leaked from a map would differ.+ let saved: Vec<Vec<u8>> = (0..4).map(|_| write_full(&Document::open(Arc::new(bytes.clone())).unwrap(), &opts).unwrap()).collect();+ assert!(saved.windows(2).all(|w| w[0] == w[1]), "full save differs between runs");+ let again = write_full(&Document::open(Arc::new(saved[0].clone())).unwrap(), &opts).unwrap();+ assert_eq!(again, write_full(&Document::open(Arc::new(saved[0].clone())).unwrap(), &opts).unwrap());+ let inc: Vec<Vec<u8>> = (0..4).map(|_| write_incremental(&Document::open(Arc::new(bytes.clone())).unwrap(), &opts).unwrap()).collect();+ assert!(inc.windows(2).all(|w| w[0] == w[1]), "incremental save differs between runs");+ if bytes == af_fixture() {+ assert!(af_survives(saved[0].clone()) && af_survives(inc[0].clone()));+ }+ }+ }+}++#[test]+fn a_2_0_document_writes_text_as_utf8() {+ let mut doc = Document::open(Arc::new(af_fixture())).unwrap();+ assert_eq!(doc.text_string("Caf\u{e9}").bytes, [&[0xEF, 0xBB, 0xBF][..], "Caf\u{e9}".as_bytes()].concat());+ doc.set_version("1.7");+ assert_eq!(doc.text_string("Caf\u{e9}").bytes[..2], [0xFE, 0xFF]);+}@@ -53,6 +53,12 @@ impl Session { if let Some(c) = self.clock { return Some(c()); }+ // The reproducible-builds convention: a set variable fixes every stamp, so that the+ // same edits to the same unencrypted file save to the same bytes.+ #[cfg(not(target_arch = "wasm32"))]+ if let Some(t) = source_date_epoch(std::env::var("SOURCE_DATE_EPOCH").ok().as_deref()) {+ return Some(t);+ } #[cfg(not(target_arch = "wasm32"))] return std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).ok().map(|d| d.as_secs() as i64); #[cfg(target_arch = "wasm32")]@@ -64,7 +70,7 @@ impl Session { /// A dynamic stamp's second line: "By Ada at 2:14 pm, Oct 02, 2026" (local time). pub fn stamp_by_line(&self, author: &str) -> String { let offset = self.local_offset();- let d = self.now().map(|t| pdfcraft_cos::pdf_date(t + offset)).unwrap_or_default();+ let d = self.now().map(|t| pdfcraft_cos::pdf_date(t.saturating_add(offset))).unwrap_or_default(); let num = |a: usize, b: usize| d.get(a..b).and_then(|s| s.parse::<u32>().ok()).unwrap_or(0); let (y, mo, day, hh, mm) = (num(2, 6), num(6, 8), num(8, 10), num(10, 12), num(12, 14)); const MONTHS: [&str; 12] = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];@@ -75,7 +81,7 @@ impl Session { pub fn today(&self) -> (i64, u32, u32) { let offset = self.local_offset();- self.now().map(|t| pdfcraft_cos::pdf_date(t + offset)).as_deref().and_then(parse_ymd).unwrap_or((1970, 1, 1))+ self.now().map(|t| pdfcraft_cos::pdf_date(t.saturating_add(offset))).as_deref().and_then(parse_ymd).unwrap_or((1970, 1, 1)) } /// Open a document from bytes. Rendering starts lazily when pages are requested.@@ -302,3 +308,21 @@ impl Session { &self.docs } }++/// `SOURCE_DATE_EPOCH` as seconds, or `None` when it is unset or not a whole number.+fn source_date_epoch(value: Option<&str>) -> Option<i64> {+ value?.trim().parse().ok()+}++#[cfg(test)]+mod source_date_tests {+ use super::source_date_epoch;++ #[test]+ fn only_a_whole_number_fixes_the_clock() {+ assert_eq!(source_date_epoch(Some(" 1700000000\n")), Some(1_700_000_000));+ for bad in [None, Some(""), Some("yesterday"), Some("1.5")] {+ assert_eq!(source_date_epoch(bad), None);+ }+ }+}@@ -13,7 +13,7 @@ use std::collections::{HashMap, HashSet}; -use pdfcraft_cos::{Dict, Document, ObjRef, Object, PdfString};+use pdfcraft_cos::{Dict, Document, ObjRef, Object}; use crate::{OrganizeError, walk}; @@ -556,7 +556,7 @@ pub fn add_bookmark_at( let parent = container(doc, parent_path)?; let mut kids = children_of(doc, parent, &mut HashSet::from([parent])); let mut d = Dict::new();- d.set(b"Title".to_vec(), Object::String(PdfString::text(title)));+ d.set(b"Title".to_vec(), Object::String(doc.text_string(title))); d.set(b"Dest".to_vec(), dest); let item = doc.add(Object::Dict(d)); let at = index.min(kids.len());@@ -593,14 +593,14 @@ pub fn add_bookmark_tree(doc: &mut Document, parent_title: &str, entries: &[Outl .map(|e| pages.get(e.page).map(|p| p.0).ok_or(OrganizeError::NoSuchPage(e.page))) .collect::<std::result::Result<Vec<_>, _>>()?; let mut d = Dict::new();- d.set(b"Title".to_vec(), Object::String(PdfString::text(parent_title)));+ d.set(b"Title".to_vec(), Object::String(doc.text_string(parent_title))); let parent = doc.add(Object::Dict(d)); // (level, item, its children so far); the bottom is the new parent. let mut stack: Vec<(u8, ObjRef, Vec<ObjRef>)> = vec![(0, parent, Vec::new())]; let mut done: Vec<(ObjRef, Vec<ObjRef>)> = Vec::new(); for (e, page) in entries.iter().zip(targets) { let mut d = Dict::new();- d.set(b"Title".to_vec(), Object::String(PdfString::text(&e.title)));+ d.set(b"Title".to_vec(), Object::String(doc.text_string(&e.title))); d.set(b"Dest".to_vec(), go_to(page)); if let Some(se) = e.element { d.set(b"SE".to_vec(), Object::Ref(se));@@ -636,7 +636,7 @@ pub fn rename_bookmark(doc: &mut Document, path: &[usize], title: &str) -> Resul return Err(OutlineError::EmptyTitle); } let (r, _) = resolve_path(doc, path)?;- put(doc, r, b"Title", Some(Object::String(PdfString::text(title))))+ put(doc, r, b"Title", Some(Object::String(doc.text_string(title)))) } /// Set the colour of a bookmark's title (`/C`, ISO 32000-2 §12.3.3, Table 151: "An array of three@@ -3767,3 +3767,19 @@ fn bookmark_style_and_colour_are_set_as_the_table_says() { let got: Vec<_> = bookmarks(&saved).into_iter().map(|b| (b.color, b.bold, b.italic)).collect(); assert_eq!(got, [(Some([0.0, 0.0, 1.0]), true, true), (None, false, false)]); }++#[test]+fn bookmark_titles_follow_the_document_version() {+ let mut doc = doc_a();+ doc.set_version("2.0");+ add_bookmark(&mut doc, &[], 0, "Caf\u{e9}", 0).unwrap();+ let raw = |d: &Document| {+ d.get(bookmarks(d)[0].obj).as_dict().and_then(|m| m.get(b"Title").and_then(|t| t.as_string()).map(|s| s.bytes.clone())).unwrap()+ };+ assert_eq!(raw(&doc)[..3], [0xEF, 0xBB, 0xBF]);+ assert_eq!(titles(&bookmarks(&doc)), ["Caf\u{e9}"]);+ let mut old = doc_a();+ old.set_version("1.7");+ add_bookmark(&mut old, &[], 0, "Caf\u{e9}", 0).unwrap();+ assert_eq!(raw(&old)[..2], [0xFE, 0xFF]);+}@@ -403,6 +403,7 @@ update the numbers. | Date | Change | Summary | |---|---|---|+| 2026-10-27 | minor | Gap 22: saves are tested byte-identical across runs and keep `/AF`; `SOURCE_DATE_EPOCH` fixes the engine clock on non-wasm targets; bookmark titles are UTF-8 in 2.0 files. Linearization, Arlington, DPart and namespaces remain. Estimate unchanged. | | 2026-10-25 | minor | Gaps 5 and 14 current state: native release builds embed LXGW WenKai Lite for Simplified Chinese, while the web build has no Hans face; Arabic catalogs stay logical with explicit RTL label direction and mirrored stock chrome, while native/browser RTL input, IME and complex workflows remain unqualified. Estimates unchanged. | | 2026-10-19 | minor | Gap 2: a Type 0 font whose `/ToUnicode` is the name `/Identity-H` or `/Identity-V` (WSCAD) reads its codes as Unicode, so such pages have text to select, find and edit (hayro-interpret patch 44, `view.render-cid-fonts`). Estimates unchanged. | | 2026-10-15 | minor | Gap 21: the side-by-side view had already shipped; the Compare panel filters text changes and visual differences. Estimate 10–20 h → 8–16 h |@@ -26,7 +26,7 @@ are borrowed; PDF 2.0 additions and most subset standards are missing. **150–2 | §7.6.4 | Standard security handler R2–R6 (RC4 40/128, AES-128, AES-256), crypt filters, `/EncryptMetadata`, SASLprep | full | **own**, read and write | `crates/crypt`; written files open in Acrobat DC since #836 (#774: Acrobat needs the catalog outside object streams) | | §7.6.5 | Public-key security handler | full | none | `core.open-public-key`, `protect.certificate-encryption` planned | | §7.6.2 | Unencrypted signature `/Contents` | yes | yes | signing encrypted documents (#753) |-| §7.9.2 | Text strings incl. PDF 2.0 UTF-8 | full | PDFDocEncoding and UTF-16; UTF-8 none | `core.pdf20-utf8-strings` planned |+| §7.9.2 | Text strings incl. PDF 2.0 UTF-8 | full | PDFDocEncoding, UTF-16 and UTF-8 read; UTF-8 written by `Document::text_string` for 2.0 files, bookmark titles only | `core.pdf20-utf8-strings` partial | | §7.11 | File specifications, embedded files | full | read and save attachments; no add/delete/describe | `organize.attachments-*` planned | | §7.12 | Extensions dictionary | yes | preserved | | | §8 | Graphics: paths, colour spaces, patterns, shadings, images, XObjects | full | **borrowed** renderer; own content parser and serializer (`crates/content`) for editing and redaction | rendering P0s partial |@@ -50,7 +50,7 @@ are borrowed; PDF 2.0 additions and most subset standards are missing. **150–2 | §14.10 | Web capture | yes | none | | | §14.11 | Prepress: page boxes, output intents, trapping, OPI | full | page boxes and crop; sRGB output intent | `print.*` production planned | | §14.12 | Document parts (DPart) | yes | none | `core.pdf20-dpart` planned |-| §14.13 | Associated files (`/AF`) | yes | none | `core.pdf20-associated-files` planned |+| §14.13 | Associated files (`/AF`) | yes | kept by every save; no listing or editing | `core.pdf20-associated-files` partial | | — | Structural validation (Arlington PDF model) | yes (Preflight syntax checks) | none | `core.arlington-validation` planned | | — | Optional content (layers) | full | **own** toggle and default state, reset to the default state; set-OCG-state actions; no layer editing | `view.layers-panel`, `view.layers-reset` | | — | Redaction (`/Redact` annotations, applying) | full | **own**: removes glyphs, image pixels, vectors, XObject content, annotations, fields and tags; verifies no residue | `protect.redact-*`, `protect.redaction-verification` |@@ -130,6 +130,7 @@ are borrowed; PDF 2.0 additions and most subset standards are missing. **150–2 | Date | Change | Summary | |---|---|---|+| 2026-10-27 | minor | §7.9.2 UTF-8 writing for 2.0 files (bookmark titles) and §14.13 `/AF` kept across saves: both partial | | 2026-10-14 | minor | §12.4.4: `/Trans` and `/Dur` read into `DocInfo::transitions` and reported by `doc_info` | | 2026-10-11 | minor | §7.6.4: encrypted files we write open in Acrobat (#774, fixed by #836) | | 2026-10-10 | major | Created from the code, `vendor/README.md` and `parity/acrobat-features.toml`, against ISO 32000-2 and Acrobat Pro 26.002.21931's documented capabilities |@@ -355,11 +355,11 @@ title = "Deterministic full-save mode" area = "A" tier = "P2" milestone = "M1"-status = "planned"+status = "partial" commands = [] tools = []-evidence = []-notes = "M1.9 option"+evidence = ["crates/cos/tests/it/writer.rs::saves_are_byte_identical_across_runs_and_keep_associated_files"]+notes = "Same document and edits give the same bytes: full and incremental saves are checked byte-identical across separately opened documents (own hash state), with and without object streams (crates/cos/tests/it/writer.rs); /ID derives from the content. SOURCE_DATE_EPOCH, when set to whole seconds, fixes the clock the engine stamps /ModDate and edit dates with. Missing: encrypted saves (random salts and IVs), a UI or automation switch, no check against another producer." [[feature]] id = "core.linearization"@@ -415,11 +415,11 @@ title = "PDF 2.0 UTF-8 text strings" area = "A" tier = "P1" milestone = "M2"-status = "planned"+status = "partial" commands = [] tools = []-evidence = []-notes = ""+evidence = ["crates/cos/tests/it/writer.rs::a_2_0_document_writes_text_as_utf8"]+notes = "Reading UTF-8 (BOM) text strings already worked. Document::text_string writes UTF-8 with a BOM for a document whose header says 2.0 and UTF-16BE before; only bookmark titles use it so far. Missing: other writers (annotation contents, Info, form values) still write UTF-16BE, which PDF 2.0 readers accept; /Version in the catalog is not consulted; not checked with veraPDF or Acrobat." [[feature]] id = "core.pdf20-associated-files"@@ -427,11 +427,11 @@ title = "PDF 2.0 associated files (/AF)" area = "A" tier = "P1" milestone = "M2"-status = "planned"+status = "partial" commands = [] tools = []-evidence = []-notes = ""+evidence = ["crates/cos/tests/it/writer.rs::saves_are_byte_identical_across_runs_and_keep_associated_files"]+notes = "Full and incremental saves keep /AF on the catalog, a page and the file specification with its embedded stream (tested). Missing: listing or editing associated files and /AFRelationship in the UI or a tool, a sanitizer choice, PDF/A-3 checks beyond preflight's AFRelationship rule." [[feature]] id = "core.pdf20-dpart"