collective
Browse / github.com/storytold/pdfcraft / patch
M7: per-cluster ActualText and per-glyph ToUnicode for edited shaped lines (#4919)
Edit text: an edited Thai line can only be selected and copied whole, and Foxit pastes its marks as blanks ## What happens After **Edit text** changes a Thai line drawn in the PDF's own complete TrueType font, the saved line can no longer be selected or copied word by word: | Reader | Select only `น้ำ` and copy | |---|---| | Acrobat | Only the whole line can be selected; the highlight covers only the first cluster (`ที่`) | | PdfCraft | Only the whole line; Find `น้ำ` highlights the whole line | | Poppler (`pdftotext -x … -W …` over one word) | The whole line | | Foxit | Copies, but pastes blanks into Word | | macOS Preview | Correct | Before the edit, the same line selects and copies word by wo
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- bomu-bomu (imported from pull request #4920, open)
- Base
1dd2737b47a9c1c9562f0af563053d1bf510e4bf- Size
- +466 −11 in 8 files
- Review
- pending
- Shared
- 2026-10-28
- Id
6d3f0fd3a1576769151a7112
How it went
No other operator has reported using it yet.
Diff
raw · ROADMAP.md crates/automation/tests/it/automation.rs crates/edit/src/tests.rs crates/edit/src/text_regions/styled.rs crates/fonts/src/source_shaping.rs docs/editing-existing-text.md docs/roadmap.md parity/acrobat-features.toml
@@ -199,6 +199,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md ## Progress log +- **2026-10-27 (M7, refs #1384):** Edit text writes a reshaped left-to-right line (Thai, Latin with combining marks) with an `/ActualText` span per cluster of several glyphs instead of one around the whole line, and gives each glyph its own character in ToUnicode where the cluster has as many glyphs as characters. A single word of an edited Thai line can be selected, found and copied again in Acrobat, Poppler and PdfCraft, and Foxit no longer pastes blanks for its marks. Right-to-left runs keep one span in typing order. Estimates unchanged. - **2026-10-24 (M10, #4158 partial):** Word, HTML, RTF and PowerPoint export keep a block that continues another on its line (an italic case name, a link, a superscript) as a run in its own style inside that paragraph, with the source's word space, instead of a paragraph of its own. Style changes that wrap onto the next line still split paragraphs (the reading blocks keep style boundaries). Estimates unchanged. - **2026-10-24 (M3, window move on Wayland, #3970 follow-up):** After a move or resize by PdfCraft's own title bar or edges, the button release given back to egui goes before the pointer leaving the window, so hover no longer sticks at the old place until the pointer moves. Found by CodeRabbit in review of #3970; a unit test covers it. Estimates unchanged. - **2026-10-24 (M14, #4253):** The annotation quick bar remembers Keep tool selected across restarts, including unpinning. Old and malformed settings keep safe defaults; the opt-in UI control channel reads and changes the same preference. Estimates unchanged.@@ -7851,6 +7851,8 @@ fn command_batch_tells_a_tool_name_from_an_unknown_or_toolless_command() { ); } +/// Thai added through the tools survives save and reopen, extracts as typed, renders and can be+/// updated; with no document Thai face, Add Text refuses it and adds nothing. #[test] fn thai_text_through_tools_saves_reopens_extracts_and_renders() { let dir = workdir("thai-content");@@ -7875,6 +7877,31 @@ fn thai_text_through_tools_saves_reopens_extracts_and_renders() { assert!(page_text(&mut a, reopened)[0].contains("แก้ไข น้ำ")); } +/// #1384: after Edit text reshapes a line, Find (and so selection and copy) still reaches a single+/// word of it, before and after saving: the line isn't one `/ActualText` unit.+#[test]+fn a_word_in_an_edited_shaped_line_is_found_on_its_own() {+ let dir = workdir("shaped-word");+ std::fs::write(dir.join("inter.ttf"), include_bytes!("../../../../assets/fonts/Inter-Regular.ttf")).unwrap();+ let mut a = auto(&dir);+ let id = ok(&mut a, "doc_open", json!({"path":"a.pdf"}))["doc"].as_u64().unwrap();+ let resource = ok(&mut a, "text_font_import", json!({"doc":id,"page":1,"path":"inter.ttf"}))["resource"].as_str().unwrap().to_owned();+ let line = "q\u{301}x\u{323} first middle last";+ // Wide enough to stay one row, so the whole line's match is one rectangle.+ ok(&mut a, "text_edit", json!({"doc":id,"page":1,"paragraph":1,"text":line,"font_resource":resource,"width":400}));+ ok(&mut a, "doc_save", json!({"doc":id,"path":"edited.pdf"}));+ let reopened = ok(&mut a, "doc_open", json!({"path":"edited.pdf"}))["doc"].as_u64().unwrap();+ for doc in [id, reopened] {+ let width = |query: &str, a: &mut _| {+ let found = ok(a, "text_find", json!({"doc":doc,"query":query}));+ let r = &found["matches"][0]["rects"][0];+ r[2].as_f64().unwrap() - r[0].as_f64().unwrap()+ };+ let (word, whole) = (width("middle", &mut a), width(line, &mut a));+ assert!(word > 0.0 && word < whole / 2.0, "middle is {word} of {whole}");+ }+}+ #[test] fn hebrew_text_through_tools_saves_reopens_extracts_and_renders() { let dir = workdir("hebrew-content");@@ -5708,6 +5708,8 @@ fn a_new_character_comes_from_the_source_fonts_own_embedded_program() { } } +/// An edited shaped run (Latin with marks, by Inter) is one text object, one `Tm` and `TJ`s only,+/// and each glyph is drawn where shaping put it, raised marks included. #[test] fn a_shaped_run_is_one_text_object_with_each_glyph_where_shaping_put_it() { let inter = include_bytes!("../../../assets/fonts/Inter-Regular.ttf");@@ -5718,9 +5720,11 @@ fn a_shaped_run_is_one_text_object_with_each_glyph_where_shaping_put_it() { let style = text::BlockStyle { width: Some(400.0), resource: Some(resource), ..Default::default() }; text::rewrite_block(&mut doc, 0, 0, Some(typed), &style).unwrap(); assert_eq!(text::text_blocks(&reopen(&doc), 0).unwrap()[0].editor_text, typed);+ // The run's text object: the one whose font is the shaped program (`PCFull…`). let ops = pdfcraft_content::parse(&page_content_bytes(&doc, 0)).ops;- let start = ops.iter().position(|o| o.is("BDC")).unwrap();- let end = start + ops[start..].iter().position(|o| o.is("EMC")).unwrap();+ let shaped_font = ops.iter().position(|o| o.is("Tf") && o.operands[0].as_name().is_some_and(|n| n.starts_with(b"PCFull"))).unwrap();+ let start = ops[..shaped_font].iter().rposition(|o| o.is("BT")).unwrap();+ let end = start + ops[start..].iter().position(|o| o.is("ET")).unwrap() + 1; let run = &ops[start..end]; let count = |name: &str| run.iter().filter(|o| o.is(name)).count(); assert_eq!((count("BT"), count("Tm"), count("Tj")), (1, 1, 0), "{run:?}");@@ -5762,6 +5766,95 @@ fn a_shaped_run_is_one_text_object_with_each_glyph_where_shaping_put_it() { assert_eq!(rise, 0.0, "the rise is reset after the run"); } +/// The `/ToUnicode` destination of every glyph the first page shows in a shaped-run font+/// (`PCFull…`), in drawing order: what a reader that ignores `/ActualText` copies.+fn shaped_glyph_destinations(doc: &Document) -> Vec<String> {+ let page = pdfcraft_model::pages(doc).remove(0);+ let resources = page.dict.get(b"Resources").and_then(|r| doc.resolve(r).as_dict().cloned()).unwrap_or_default();+ let fonts = resources.get(b"Font").and_then(|f| doc.resolve(f).as_dict().cloned()).unwrap_or_default();+ let map_of = |name: &[u8]| -> std::collections::HashMap<u16, String> {+ let font = doc.resolve(fonts.get(name).unwrap()).as_dict().unwrap().clone();+ let Object::Stream(cmap) = &*doc.resolve(font.get(b"ToUnicode").unwrap()) else { panic!("no ToUnicode") };+ let text = String::from_utf8(cmap.decoded().unwrap()).unwrap();+ text.lines()+ .filter_map(|line| {+ let (code, dest) = line.strip_prefix('<')?.split_once("> <")?;+ let units: Vec<u16> = dest+ .trim_end_matches('>')+ .as_bytes()+ .chunks(4)+ .map(|h| u16::from_str_radix(std::str::from_utf8(h).unwrap(), 16).unwrap())+ .collect();+ Some((u16::from_str_radix(code, 16).ok()?, String::from_utf16(&units).unwrap()))+ })+ .collect()+ };+ let (mut map, mut out) = (None, Vec::new());+ for op in pdfcraft_content::parse(&page_content_bytes(doc, 0)).ops {+ if op.is("Tf") {+ let name = op.operands[0].as_name().unwrap().to_vec();+ map = name.starts_with(b"PCFull").then(|| map_of(&name));+ continue;+ }+ let Some(map) = &map else { continue };+ let strings: Vec<Vec<u8>> = match op.op.as_slice() {+ b"TJ" => op.operands[0].as_array().unwrap().iter().filter_map(|o| o.as_string().map(|s| s.bytes.clone())).collect(),+ b"Tj" => vec![op.operands[0].as_string().unwrap().bytes.clone()],+ _ => continue,+ };+ for code in strings.iter().flat_map(|s| s.chunks(2)) {+ out.push(map.get(&u16::from_be_bytes([code[0], code[1]])).cloned().unwrap_or_default());+ }+ }+ out+}++/// #1384: an edited left-to-right shaped run is not wrapped in one `/ActualText`, which made+/// Acrobat and PdfCraft select only the whole line; each cluster of several glyphs gets its own+/// span instead (as Chrome writes them), and each glyph carries its own characters in+/// `/ToUnicode` for readers that ignore `/ActualText` (Foxit), except where a cluster has more+/// glyphs than characters (Sara Am drawn as two glyphs).+#[test]+fn an_edited_shaped_run_maps_each_glyph_to_its_own_characters() {+ let inter = include_bytes!("../../../assets/fonts/Inter-Regular.ttf").as_slice();+ // A static Thai face: the build's, or Sarabun from the craft-fonts checkout (the variable+ // Noto Sans Thai can't be embedded). Without one, only the Latin case runs. A relative+ // `CRAFT_FONTS_DIR` is resolved as the fonts build script reads it, from that crate's folder.+ let fonts_crate = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../fonts");+ let sarabun = option_env!("CRAFT_FONTS_DIR").and_then(|dir| std::fs::read(fonts_crate.join(dir).join("fonts/sarabun/Sarabun-Regular.ttf")).ok());+ let thai = pdfcraft_fonts::document_thai_font()+ .map(|f| f.bytes.to_vec())+ .filter(|b| pdfcraft_fonts::embedding::validate(b).is_ok())+ .or(sarabun)+ .filter(|b| pdfcraft_fonts::embedding::validate(b).is_ok());+ let hex = |s: &str| s.encode_utf16().map(|u| format!("{u:04X}")).collect::<String>();+ // A leading vowel (ใ) is a cluster of one glyph, so it needs no span.+ let cases = [+ (Some(inter), "q\u{301}x\u{323} words", ["q\u{301}", "x\u{323}"].as_slice(), true),+ (thai.as_deref(), "ที่ดิน กิ่ ผู้ใช้ ปู่", ["ที่", "ดิ", "กิ่", "ผู้", "ช้", "ปู่"].as_slice(), true),+ (thai.as_deref(), "ที่ดิน น้ำ ปู่", ["ที่", "ดิ", "น้ำ", "ปู่"].as_slice(), false),+ ];+ for (bytes, typed, clusters, every_glyph_mapped) in cases {+ let Some(bytes) = bytes else { continue };+ let mut doc = text_page("BT /F1 12 Tf 72 700 Td (Stable text) Tj ET");+ let resource = text::register_text_font(&mut doc, 0, bytes).unwrap();+ let style = text::BlockStyle { width: Some(400.0), resource: Some(resource), ..Default::default() };+ text::rewrite_block(&mut doc, 0, 0, Some(typed), &style).unwrap();+ let doc = reopen(&doc);+ assert_eq!(text::text_blocks(&doc, 0).unwrap()[0].editor_text, typed);+ let content = String::from_utf8_lossy(&page_content_bytes(&doc, 0)).into_owned();+ let spans: Vec<String> = content.split("/ActualText <FEFF").skip(1).filter_map(|s| s.split('>').next()).map(str::to_owned).collect();+ assert_eq!(spans, clusters.iter().map(|c| hex(c)).collect::<Vec<_>>(), "{typed}: {content}");+ // What a reader ignoring ActualText copies: every glyph's own character, in drawing order,+ // so two glyphs given each other's characters fail, not only empty entries.+ let destinations = shaped_glyph_destinations(&doc);+ assert_eq!(destinations.concat(), typed, "{typed}: {destinations:?}");+ if every_glyph_mapped {+ assert_eq!(destinations, typed.chars().map(String::from).collect::<Vec<_>>(), "{typed}");+ }+ }+}+ #[test] fn a_subset_program_is_not_trusted_for_new_characters_and_the_error_names_the_character() { let mut doc = embedded_program_page("BT /F1 12 Tf 72 700 Td (Stable text) Tj ET", "ABCDEF+Inter-Regular");@@ -674,9 +674,17 @@ pub(crate) fn rewrite_styled_block( block_ops.extend(state.paint_ops(after)); if let Some((_, shaped, _)) = shaped_runs.get(&(row_index, pos)) { let resource = glyph_resources.get(&(row_index, pos)).ok_or_else(|| EditError::Invalid("shaped font is missing".into()))?;- let mut properties = Dict::new();- properties.set(b"ActualText".to_vec(), Object::String(PdfString::text(&string)));- block_ops.push(Op::new("BDC", vec![Object::name("Span"), Object::Dict(properties)]));+ let actual_text = |text: &str| {+ let mut properties = Dict::new();+ properties.set(b"ActualText".to_vec(), Object::String(PdfString::text(text)));+ Op::new("BDC", vec![Object::name("Span"), Object::Dict(properties)])+ };+ // A right-to-left run is drawn in visual order, so it reads back in typing order+ // only as one span. A left-to-right run gets a span per cluster of several glyphs+ // instead, so a single word of it can still be selected and copied.+ if shaped.rtl {+ block_ops.push(actual_text(&string));+ } state.font = Some((resource.clone(), size_of(source, item.1))); // One text object for the run: each glyph is placed where shaping put it by a TJ // adjustment from where the glyph before it ends (its advance in the embedded@@ -703,7 +711,17 @@ pub(crate) fn rewrite_styled_block( // `pen`: where the next glyph would start, in thousandths of the font size. let (mut pen, mut extra, mut raised) = (0.0, 0.0, rise); let mut shown_glyphs: Vec<Object> = Vec::new();+ // Glyphs left in the open cluster span (marked-content operators may stand in a+ // text object, ISO 32000-1 §14.6, but not inside a TJ array).+ let mut in_span = 0usize; for (i, glyph) in shaped.glyphs.iter().enumerate() {+ if let Some((text, count)) = glyph.span.as_ref().filter(|_| !shaped.rtl && in_span == 0) {+ if !shown_glyphs.is_empty() {+ block_ops.push(Op::new("TJ", vec![Object::Array(std::mem::take(&mut shown_glyphs))]));+ }+ block_ops.push(actual_text(text));+ in_span = *count;+ } let target = (glyph.position[0] * size + extra) * per_unit; let glyph_rise = rise + glyph.position[1] * size; if (glyph_rise - raised).abs() > 1e-9 || shown_glyphs.len() >= 4096 {@@ -722,16 +740,37 @@ pub(crate) fn rewrite_styled_block( shown_glyphs.push(Object::String(PdfString::literal(code.to_be_bytes().to_vec()))); pen = target + shown.width(u32::from(code)) * 1000.0; extra += char_spacing * glyph.text.chars().count() as f64 + word_spacing * glyph.text.matches(' ').count() as f64;+ if in_span > 0 {+ in_span -= 1;+ if in_span == 0 {+ // Inside the span, move on to where the next glyph starts: a mark set+ // back over its letter otherwise leaves the move to the next cluster,+ // which reads as a word space.+ let next = shaped.glyphs.get(i + 1).map_or(shaped.advance, |g| g.position[0]);+ let next_target = (next * size + extra) * per_unit;+ if next_target.is_finite() && (next_target - pen).abs() > 1e-6 {+ shown_glyphs.push(n(pen - next_target));+ pen = next_target;+ }+ block_ops.push(Op::new("TJ", vec![Object::Array(std::mem::take(&mut shown_glyphs))]));+ block_ops.push(Op::new("EMC", vec![]));+ }+ } } if !shown_glyphs.is_empty() { block_ops.push(Op::new("TJ", vec![Object::Array(shown_glyphs)])); }+ if in_span > 0 {+ block_ops.push(Op::new("EMC", vec![]));+ } if (raised - rise).abs() > 1e-9 { block_ops.push(Op::new("Ts", vec![n(rise)])); } block_ops.push(Op::new("ET", vec![])); (state.char_spacing, state.word_spacing) = (char_spacing, word_spacing);- block_ops.push(Op::new("EMC", vec![]));+ if shaped.rtl {+ block_ops.push(Op::new("EMC", vec![]));+ } segment_width = (shaped.advance * size + extra) * state.scale * source.origin.k; } else { block_ops.push(Op::new("BT", vec![]));@@ -1,19 +1,29 @@ //! Shaping an explicitly selected source/document face, with OpenType features and UAX #9. use harfrust::{BufferClusterLevel, Direction, ShapeOptions, ShaperData, UnicodeBuffer}; use skrifa::raw::TableProvider;-use skrifa::{FontRef, GlyphId};+use skrifa::{FontRef, GlyphId, MetadataProvider}; use unicode_bidi::{Level, ParagraphBidiInfo}; #[derive(Clone, Debug)] pub struct ShapedGlyph { pub id: GlyphId, pub position: [f64; 2],+ /// The characters this glyph stands for in `/ToUnicode`: its own in a left-to-right cluster+ /// whose glyphs each take one, else the whole cluster on its first glyph and none on the rest. pub text: String,+ /// On the first glyph of a left-to-right cluster of several glyphs (a letter and its marks,+ /// Sara Am drawn as two glyphs): the cluster's text and glyph count, for an `/ActualText`+ /// span around just that cluster, as Chrome writes them. Readers that honour it read the+ /// marks in typing order wherever shaping put them; the rest read `text`.+ pub span: Option<(String, usize)>, } #[derive(Clone, Debug)] pub struct ShapedRun { pub glyphs: Vec<ShapedGlyph>, pub advance: f64,+ /// Whether any of it is right-to-left: drawn in visual order, it then needs one `/ActualText`+ /// in typing order around the whole run.+ pub rtl: bool, } /// Complex scripts/marks require shaping instead of independently encoding characters.@@ -22,12 +32,17 @@ pub fn required(text: &str) -> bool { .any(|c| matches!(u32::from(c), 0x0300..=0x036f | 0x0590..=0x0fff | 0x1780..=0x17ff | 0x200c..=0x200f | 0x202a..=0x202e | 0x2066..=0x2069)) } +/// Shape one row of `text` with the face in `bytes`, run by run in visual (bidi) order: each glyph+/// with its position in ems and the characters it stands for in `/ToUnicode`, its left-to-right+/// clusters marked for their own `/ActualText` spans ([`ShapedGlyph::span`]). Errors on a row+/// over 256 KiB or with a line break, a face that can't be read, Arabic or Khmer in a face with no+/// `GSUB`, and a glyph the face lacks. pub fn shape(bytes: &[u8], text: &str) -> Result<ShapedRun, String> { if text.len() > 262_144 || text.contains(['\n', '\r']) { return Err("shape one bounded text row at a time".into()); } if text.is_empty() {- return Ok(ShapedRun { glyphs: Vec::new(), advance: 0.0 });+ return Ok(ShapedRun { glyphs: Vec::new(), advance: 0.0, rtl: false }); } let face = FontRef::new(bytes).map_err(|e| e.to_string())?; if text.chars().any(|c| matches!(u32::from(c), 0x0600..=0x06ff | 0x0750..=0x077f | 0x08a0..=0x0fff | 0x1780..=0x17ff)) && face.gsub().is_err() {@@ -39,11 +54,15 @@ pub fn shape(bytes: &[u8], text: &str) -> Result<ShapedRun, String> { let base = if unicode_bidi::get_base_direction(text) == unicode_bidi::Direction::Rtl { Level::rtl() } else { Level::ltr() }; let bidi = ParagraphBidiInfo::new(text, Some(base)); let (levels, runs) = bidi.visual_runs(0..text.len());- let mut glyphs = Vec::new();+ let charmap = face.charmap();+ let mut glyphs: Vec<ShapedGlyph> = Vec::new(); let mut x = 0.0;+ let mut any_rtl = false; for run in runs { let Some(input) = text.get(run.clone()) else { continue }; let rtl = levels.get(run.start).is_some_and(Level::is_rtl);+ any_rtl |= rtl;+ let run_start = glyphs.len(); let mut buffer = UnicodeBuffer::new(); buffer.push_str(input); buffer.set_cluster_level(BufferClusterLevel::MonotoneGraphemes);@@ -65,12 +84,281 @@ pub fn shape(bytes: &[u8], text: &str) -> Result<ShapedRun, String> { id: GlyphId::new(g.glyph_id), position: [x + f64::from(p.x_offset) / em, f64::from(p.y_offset) / em], text: logical,+ span: None, }); x += f64::from(p.x_advance) / em; }+ if !rtl {+ let clusters: Vec<u32> = output.glyph_infos().iter().map(|g| g.cluster).collect();+ split_clusters(glyphs.get_mut(run_start..).unwrap_or_default(), &clusters, |c| charmap.map(c));+ } } if !x.is_finite() || glyphs.len() > 65_534 { return Err("shaped row exceeds its geometry budget".into()); }- Ok(ShapedRun { glyphs, advance: x })+ Ok(ShapedRun { glyphs, advance: x, rtl: any_rtl })+}++/// Mark each left-to-right cluster of several glyphs for a span, and give each of its glyphs its+/// own character when the cluster has as many glyphs as characters and they correspond in order:+/// every glyph is its character's own (the face's cmap) and stands at its character's place, or+/// the cluster is Thai, whose shaping substitutes mark forms one for one (Sara Am, split into two+/// glyphs, has one more glyph than characters), and at most one of its glyphs is such a form, so+/// the others fix where every character is drawn. Otherwise the text stays on the first glyph. `glyphs` and+/// `clusters` (harfrust's cluster of each) are one run, in order; each cluster's text is already+/// on its first glyph.+fn split_clusters(glyphs: &mut [ShapedGlyph], clusters: &[u32], nominal: impl Fn(char) -> Option<GlyphId>) {+ let mut at = 0;+ while at < glyphs.len() {+ let cluster = clusters.get(at).copied();+ let len = clusters.get(at..).unwrap_or_default().iter().take_while(|c| Some(**c) == cluster).count().max(1);+ let Some(group) = glyphs.get_mut(at..at + len) else { break };+ at += len;+ if group.len() < 2 {+ continue;+ }+ let text = group.first().map(|g| g.text.clone()).unwrap_or_default();+ let chars: Vec<char> = text.chars().collect();+ let thai = chars.iter().all(|c| matches!(u32::from(*c), 0x0e01..=0x0e5b));+ // Shaping sorts a cluster's marks by combining class (`กฺุ` is drawn with ุ before ฺ, and+ // a tone typed before a lower vowel after it), so the drawn order has to be shown, not+ // assumed: every glyph that is some character's own must stand at that character's place,+ // and at most one may be a substituted form (a lowered tone mark), which is no character's+ // own. The placed glyphs then leave it only its own place; two substituted forms could+ // have been swapped. The characters' own glyphs are collected once, so a cluster of many+ // marks stays linear.+ let one_for_one = chars.len() == group.len() && {+ let own: Vec<Option<GlyphId>> = chars.iter().map(|c| nominal(*c)).collect();+ let owned: std::collections::HashSet<GlyphId> = own.iter().flatten().copied().collect();+ let placed = group.iter().zip(&own).all(|(g, o)| *o == Some(g.id) || !owned.contains(&g.id));+ let substituted = group.iter().filter(|g| !owned.contains(&g.id)).count();+ placed && substituted <= usize::from(thai)+ };+ if one_for_one {+ for (glyph, c) in group.iter_mut().zip(chars) {+ glyph.text = c.to_string();+ }+ }+ if let Some(first) = group.first_mut() {+ first.span = Some((text, len));+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ /// A cluster's glyphs as shaping leaves them: its text on the first glyph, none on the rest.+ fn cluster(text: &str, ids: &[u32]) -> Vec<ShapedGlyph> {+ ids.iter()+ .enumerate()+ .map(|(i, id)| ShapedGlyph {+ id: GlyphId::new(*id),+ position: [0.0, 0.0],+ text: if i == 0 { text.to_owned() } else { String::new() },+ span: None,+ })+ .collect()+ }++ /// #4919: each glyph of a cluster takes its own character when they correspond one for one,+ /// and every cluster of several glyphs is marked for its own span. No font is needed: the+ /// face's cmap is a function from characters to glyphs.+ #[test]+ fn clusters_map_each_glyph_to_its_character_where_they_correspond() {+ // The face's own glyphs: ก 10, ท 11, น 12, ป 13, ี 20, ่ 21, ้ 22, า 23, ํ 24, ุ 25, ู 26, ฺ 27,+ // q 30, ́ 31, a 32, ̣ 33, n 34, ̃ 35, ̄ 36, ๛ 37 (the last Thai character), U+0E5C 38.+ let nominal = |c: char| {+ let id = match c {+ 'ก' => 10,+ 'ท' => 11,+ 'น' => 12,+ 'ป' => 13,+ '\u{e35}' => 20,+ '\u{e48}' => 21,+ '\u{e49}' => 22,+ '\u{e32}' => 23,+ '\u{e4d}' => 24,+ '\u{e38}' => 25,+ '\u{e39}' => 26,+ '\u{e3a}' => 27,+ 'q' => 30,+ '\u{301}' => 31,+ 'a' => 32,+ '\u{323}' => 33,+ 'n' => 34,+ '\u{303}' => 35,+ '\u{304}' => 36,+ '\u{e5b}' => 37,+ '\u{e5c}' => 38,+ _ => return None,+ };+ Some(GlyphId::new(id))+ };+ let texts = |glyphs: &[ShapedGlyph]| glyphs.iter().map(|g| g.text.clone()).collect::<Vec<_>>();+ let spans = |glyphs: &[ShapedGlyph]| glyphs.iter().map(|g| g.span.clone()).collect::<Vec<_>>();+ let cases: [(&str, &[u32], &[&str]); 17] = [+ // Thai, with the tone mark swapped for a lowered form (99): still one for one.+ ("ที่", &[11, 20, 99], &["ท", "\u{e35}", "\u{e48}"]),+ ("ปู่", &[13, 26, 99], &["ป", "\u{e39}", "\u{e48}"]),+ // Marks shaping reordered by combining class: ุ drawn before ฺ, and a tone typed before+ // its lower vowel drawn after it. The cluster stays on the first glyph.+ ("ก\u{e3a}\u{e38}", &[10, 25, 27], &["ก\u{e3a}\u{e38}", "", ""]),+ ("ป\u{e48}\u{e39}", &[13, 26, 99], &["ป\u{e48}\u{e39}", "", ""]),+ // Sara Am split into nikhahit and sara aa: one more glyph than characters.+ ("น้ำ", &[12, 24, 22, 23], &["น้ำ", "", "", ""]),+ // Latin with the face's own mark glyph.+ ("q\u{301}", &[30, 31], &["q", "\u{301}"]),+ // Latin whose mark was substituted (a stacking form): the glyphs can't be told apart.+ ("q\u{301}", &[30, 98], &["q\u{301}", ""]),+ // Devanagari reordered (the i-matra drawn before its consonant): not split.+ ("कि", &[41, 40], &["कि", ""]),+ // A ligature: one glyph for two characters, no span needed.+ ("fi", &[50], &["fi"]),+ // A letter on its own.+ ("ก", &[10], &["ก"]),+ // More characters than glyphs (ñ composed, then the macron): the text stays on the first.+ ("n\u{303}\u{304}", &[80, 36], &["n\u{303}\u{304}", ""]),+ // Latin marks drawn swapped, both the face's own glyphs: not in place, not split.+ ("a\u{301}\u{323}", &[32, 33, 31], &["a\u{301}\u{323}", "", ""]),+ ("a\u{301}\u{323}", &[32, 31, 33], &["a", "\u{301}", "\u{323}"]),+ // Thai with a non-Thai mark is not Thai: own glyphs in place map, a substitution doesn't.+ ("ก\u{301}", &[10, 31], &["ก", "\u{301}"]),+ ("ก\u{301}", &[10, 98], &["ก\u{301}", ""]),+ // The Thai range ends at U+0E5B: one substitution is allowed there, not past it.+ ("ก\u{e5b}", &[10, 98], &["ก", "\u{e5b}"]),+ ("ก\u{e5c}", &[10, 98], &["ก\u{e5c}", ""]),+ ];+ for (text, ids, expected) in cases {+ let mut glyphs = cluster(text, ids);+ let clusters = vec![7; glyphs.len()];+ split_clusters(&mut glyphs, &clusters, nominal);+ assert_eq!(texts(&glyphs), expected.iter().map(|s| s.to_string()).collect::<Vec<_>>(), "{text}");+ let span = (ids.len() > 1).then(|| (text.to_owned(), ids.len()));+ let mut want = vec![None; ids.len()];+ want[0] = span;+ assert_eq!(spans(&glyphs), want, "{text}");+ }+ // Clusters are found by harfrust's cluster values, one run after another.+ let mut glyphs = cluster("ก", &[10]);+ glyphs.extend(cluster("ที่", &[11, 20, 21]));+ split_clusters(&mut glyphs, &[0, 3, 3, 3], nominal);+ assert_eq!(texts(&glyphs), ["ก", "ท", "\u{e35}", "\u{e48}"]);+ assert_eq!(spans(&glyphs), [None, Some(("ที่".to_owned(), 3)), None, None]);+ }++ /// The same with real shaping, by every Thai face this build embeds: the interface face (Noto+ /// Sans Thai, in every build with craft-fonts, CI's included) and the document face. Shaping+ /// reads the font only, so a variable face does. The marks shaping reorders keep their cluster+ /// on the first glyph; the rest map one for one.+ #[test]+ fn real_thai_shaping_maps_glyphs_only_in_typing_order() {+ let mut faces = crate::ui_thai_fonts();+ faces.extend(crate::document_thai_font().filter(|d| !faces.iter().any(|f| std::ptr::eq(*f, *d))));+ for face in faces {+ let texts = |text: &str| shape(face.bytes, text).unwrap().glyphs.into_iter().map(|g| g.text).collect::<Vec<_>>();+ let name = face.family;+ assert_eq!(texts("ปู่"), ["ป", "\u{e39}", "\u{e48}"], "{name}");+ assert_eq!(texts("กิ่"), ["ก", "\u{e34}", "\u{e48}"], "{name}");+ assert_eq!(texts("ป\u{e48}\u{e39}"), ["ป\u{e48}\u{e39}", "", ""], "{name}");+ assert_eq!(texts("ก\u{e3a}\u{e38}"), ["ก\u{e3a}\u{e38}", "", ""], "{name}");+ }+ }++ /// Property: whatever order shaping draws a Thai cluster's marks in, and whichever of them it+ /// swaps for another form, a cluster mapped glyph by glyph gives every glyph its own character.+ /// Every sequence of one to three marks after a letter, in every drawn order, with every set+ /// of substituted glyphs (ids no character owns), the letter's included. A cluster it can't map stays on its first+ /// glyph, which is never wrong.+ #[test]+ fn a_thai_cluster_is_mapped_glyph_by_glyph_only_when_every_glyph_is_placed() {+ const MARKS: [char; 9] = ['\u{e38}', '\u{e39}', '\u{e3a}', '\u{e34}', '\u{e35}', '\u{e48}', '\u{e49}', '\u{e47}', '\u{e4c}'];+ let nominal = |c: char| {+ Some(GlyphId::new(match c {+ 'ก' => 10,+ c => 100 + u32::from(c) - 0x0e00,+ }))+ };+ let mut sequences: Vec<Vec<char>> = vec![Vec::new()];+ for _ in 0..3 {+ let longer: Vec<Vec<char>> =+ sequences.iter().filter(|s| s.len() < 3).flat_map(|s| MARKS.iter().map(move |m| [s.as_slice(), &[*m]].concat())).collect();+ sequences.extend(longer);+ }+ /// Every order of `rest`: the drawn orders shaping could give a cluster's marks.+ fn permutations(rest: Vec<usize>) -> Vec<Vec<usize>> {+ if rest.is_empty() {+ return vec![Vec::new()];+ }+ let mut all = Vec::new();+ for (k, first) in rest.iter().enumerate() {+ let mut others = rest.clone();+ others.remove(k);+ all.extend(permutations(others).into_iter().map(|tail| [vec![*first], tail].concat()));+ }+ all+ }+ let mut checked = 0;+ for marks in sequences.iter().filter(|s| !s.is_empty()) {+ let chars: Vec<char> = std::iter::once('ก').chain(marks.iter().copied()).collect();+ let text: String = chars.iter().collect();+ for order in permutations((0..marks.len()).collect()) {+ // The letter stays first; the marks are drawn in `order`.+ let drawn: Vec<char> = std::iter::once('ก').chain(order.iter().map(|i| marks[*i])).collect();+ for substituted in 0u32..(1 << (marks.len() + 1)) {+ let ids: Vec<u32> = drawn+ .iter()+ .enumerate()+ .map(|(i, c)| match i {+ i if substituted & (1 << i) != 0 => 900 + i as u32,+ 0 => 10,+ _ => nominal(*c).map_or(0, |g| g.to_u32()),+ })+ .collect();+ let mut glyphs = cluster(&text, &ids);+ split_clusters(&mut glyphs, &vec![0; ids.len()], nominal);+ if glyphs.iter().skip(1).any(|g| !g.text.is_empty()) {+ for (glyph, truth) in glyphs.iter().zip(&drawn) {+ assert_eq!(glyph.text, truth.to_string(), "{text:?} drawn {drawn:?}, substituted {substituted:b}");+ }+ } else {+ assert_eq!(glyphs[0].text, text);+ }+ checked += 1;+ }+ }+ }+ assert!(checked > 60_000, "{checked}");+ }++ /// Malformed runs never panic: no glyphs, and fewer cluster values than glyphs (the glyphs past+ /// them are left as shaped).+ #[test]+ fn split_clusters_takes_empty_and_short_cluster_lists() {+ let nominal = |_| None;+ split_clusters(&mut [], &[], nominal);+ let mut glyphs = cluster("ที่", &[11, 20, 21]);+ split_clusters(&mut glyphs, &[], nominal);+ split_clusters(&mut glyphs, &[5], nominal);+ assert_eq!(glyphs.iter().map(|g| g.text.as_str()).collect::<Vec<_>>(), ["ที่", "", ""]);+ }++ /// A cluster of many marks (one letter with 20,000 tone marks, within the input limit) looks+ /// each character's own glyph up once, not once per glyph, so it can't stall shaping.+ #[test]+ fn a_cluster_of_many_marks_looks_each_character_up_once() {+ let text: String = std::iter::once('ก').chain(std::iter::repeat_n('\u{e48}', 20_000)).collect();+ let mut ids = vec![10];+ ids.extend(std::iter::repeat_n(21, 20_000));+ let mut glyphs = cluster(&text, &ids);+ let clusters = vec![0; glyphs.len()];+ let lookups = std::cell::Cell::new(0usize);+ split_clusters(&mut glyphs, &clusters, |c| {+ lookups.set(lookups.get() + 1);+ Some(GlyphId::new(if c == 'ก' { 10 } else { 21 }))+ });+ assert_eq!(glyphs.last().map(|g| g.text.as_str()), Some("\u{e48}"));+ assert!(lookups.get() <= ids.len(), "{} lookups for {} characters", lookups.get(), ids.len());+ } }@@ -68,6 +68,12 @@ rules, preserves marks and ligatures, and applies Unicode bidi ordering for a si Each shaped run is one text object: its glyphs are shown in one `TJ` array whose adjustments place each glyph where shaping put it, and a mark set above or below its letter is raised with the text rise. Logical text remains available after save/reopen through ToUnicode and ActualText.+A right-to-left run, drawn in visual order, has one ActualText span in typing order. A+left-to-right run has a span per cluster of several glyphs instead (a Thai letter with its+vowel and tone marks, `q́`), as Chrome writes them, so a single word can still be selected,+found and copied in Acrobat, Poppler and PdfCraft; each glyph also carries its own character in+ToUnicode where the cluster has as many glyphs as characters, for readers that ignore+ActualText (Foxit). Added Arabic text records ActualText per shaped line for copying and searching in typing order. ActualText already supplies typing order, so the editor preserves it without applying a second bidi reversal; adjacent visual glyph runs still recover their own reading order.@@ -1,5 +1,6 @@ # PdfCraft roadmap: milestones and what's next +> **Last reviewed:** 2026-10-27 · **Last updated:** 2026-10-27 · **Change:** minor (M7, refs #1384: edited left-to-right shaped lines get per-cluster ActualText and per-glyph ToUnicode, so single words of edited Thai text can be selected and copied; estimates unchanged) > **Last reviewed:** 2026-10-25 | **Last updated:** 2026-10-25 | **Change:** minor (M14 localization current state: Arabic logical catalog, explicit RTL label direction and mirrored stock chrome; native/browser RTL workflows unqualified. Native release Hans face via LXGW WenKai Lite; estimates unchanged) > **Last reviewed:** 2026-10-23 · **Last updated:** 2026-10-23 · **Change:** minor (M7, #2636 partial: pasting added text boxes and images in the engine and through `content_copy`; estimates unchanged) > **Last reviewed:** 2026-10-23 · **Last updated:** 2026-10-23 · **Change:** minor (M11: image optimization accounts for page-local UserUnit; estimates unchanged)@@ -1878,7 +1878,7 @@ milestone = "M7" status = "partial" commands = ["edit.edit_text"] tools = ["text_lines", "text_paragraphs", "text_edit", "text_edit_layout", "text_font_files", "text_font_import", "text_fonts"]-evidence = ["crates/edit/src/tests.rs::paragraphs_are_found_and_rewrapped", "crates/edit/src/tests.rs::text_lines_are_found_and_replaced_in_place", "crates/ui-egui/tests/it/editing.rs::editing_existing_text_in_place", "crates/automation/tests/it/automation.rs::editing_existing_text_through_tools", "crates/edit/src/tests.rs::a_line_whose_operator_starts_the_next_stream_can_be_edited", "crates/automation/tests/it/automation.rs::a_line_split_across_content_streams_is_listed_and_edited", "crates/ui-egui/tests/it/editing.rs::selected_text_formatting_keeps_the_rest_of_the_paragraph_and_reopens", "crates/edit/src/tests.rs::marked_paragraph_boundaries_and_cross_stream_transforms_survive_rewrite", "crates/automation/tests/it/automation.rs::explicit_font_and_disjoint_ranges_save_reopen_through_tools", "crates/edit/src/tests.rs::disjoint_style_ranges_preserve_unselected_runs_and_validate_utf8_atomically", "crates/engine/tests/it/document_fonts.rs::opentype_cff_document_font_preserves_program_codes_and_rendering_after_reopen", "crates/ui-egui/tests/it/editing.rs::cropped_images_do_not_intercept_text_and_stay_inside_each_page", "crates/automation/tests/it/automation.rs::clipped_image_bounds_and_text_edit_save_reopen_through_tools", "crates/edit/src/tests.rs::mixed_styles_and_matrix_sizes_form_visual_regions_without_merging_columns", "crates/edit/src/tests.rs::rich_region_edits_preserve_styles_positions_and_delete_all_runs_after_reopen", "crates/ui-egui/tests/it/editing.rs::one_editor_contains_mixed_styles_and_visual_rows", "crates/ui-egui/tests/it/editing.rs::inline_editor_preserves_source_font_and_previews_live_bounds", "crates/fonts/src/pdf/tests.rs::mac_subset_cmap_uses_pdf_unicode_without_changing_source_program", "crates/automation/tests/it/automation.rs::mixed_visual_region_edit_preview_save_reopen_and_undo_through_tools", "crates/ui-egui/tests/it/ui.rs::the_edit_tab_turns_on_edit_text_and_images", "crates/ui-egui/tests/it/editing.rs::the_caret_on_a_small_row_is_as_tall_as_that_row", "crates/edit/src/tests.rs::uzbek_and_kazakh_cyrillic_use_the_bundled_latin_face", "crates/automation/tests/it/automation.rs::editing_uzbek_and_kazakh_cyrillic_through_tools", "crates/edit/src/tests.rs::styled_paragraph_batches_cyrillic_without_changing_source_fonts", "crates/edit/src/tests.rs::styled_paragraph_over_240_fallback_glyphs_splits_and_refuses_atomically", "crates/edit/src/tests.rs::styled_fallback_batches_keep_face_choices_and_range_styles", "crates/edit/src/tests.rs::styled_fallback_shards_keep_text_advances_and_alignment", "crates/edit/src/tests.rs::styled_fallback_budget_is_enforced_before_objects_are_written", "crates/edit/src/tests.rs::styled_fallback_same_base_font_and_character_keep_both_weights", "crates/automation/tests/it/automation.rs::styled_cyrillic_fallback_is_batched_through_tools_and_reopen"]+evidence = ["crates/edit/src/tests.rs::paragraphs_are_found_and_rewrapped", "crates/edit/src/tests.rs::text_lines_are_found_and_replaced_in_place", "crates/ui-egui/tests/it/editing.rs::editing_existing_text_in_place", "crates/automation/tests/it/automation.rs::editing_existing_text_through_tools", "crates/edit/src/tests.rs::a_line_whose_operator_starts_the_next_stream_can_be_edited", "crates/automation/tests/it/automation.rs::a_line_split_across_content_streams_is_listed_and_edited", "crates/ui-egui/tests/it/editing.rs::selected_text_formatting_keeps_the_rest_of_the_paragraph_and_reopens", "crates/edit/src/tests.rs::marked_paragraph_boundaries_and_cross_stream_transforms_survive_rewrite", "crates/automation/tests/it/automation.rs::explicit_font_and_disjoint_ranges_save_reopen_through_tools", "crates/edit/src/tests.rs::disjoint_style_ranges_preserve_unselected_runs_and_validate_utf8_atomically", "crates/engine/tests/it/document_fonts.rs::opentype_cff_document_font_preserves_program_codes_and_rendering_after_reopen", "crates/ui-egui/tests/it/editing.rs::cropped_images_do_not_intercept_text_and_stay_inside_each_page", "crates/automation/tests/it/automation.rs::clipped_image_bounds_and_text_edit_save_reopen_through_tools", "crates/edit/src/tests.rs::mixed_styles_and_matrix_sizes_form_visual_regions_without_merging_columns", "crates/edit/src/tests.rs::rich_region_edits_preserve_styles_positions_and_delete_all_runs_after_reopen", "crates/ui-egui/tests/it/editing.rs::one_editor_contains_mixed_styles_and_visual_rows", "crates/ui-egui/tests/it/editing.rs::inline_editor_preserves_source_font_and_previews_live_bounds", "crates/fonts/src/pdf/tests.rs::mac_subset_cmap_uses_pdf_unicode_without_changing_source_program", "crates/automation/tests/it/automation.rs::mixed_visual_region_edit_preview_save_reopen_and_undo_through_tools", "crates/ui-egui/tests/it/ui.rs::the_edit_tab_turns_on_edit_text_and_images", "crates/ui-egui/tests/it/editing.rs::the_caret_on_a_small_row_is_as_tall_as_that_row", "crates/edit/src/tests.rs::uzbek_and_kazakh_cyrillic_use_the_bundled_latin_face", "crates/automation/tests/it/automation.rs::editing_uzbek_and_kazakh_cyrillic_through_tools", "crates/edit/src/tests.rs::styled_paragraph_batches_cyrillic_without_changing_source_fonts", "crates/edit/src/tests.rs::styled_paragraph_over_240_fallback_glyphs_splits_and_refuses_atomically", "crates/edit/src/tests.rs::styled_fallback_batches_keep_face_choices_and_range_styles", "crates/edit/src/tests.rs::styled_fallback_shards_keep_text_advances_and_alignment", "crates/edit/src/tests.rs::styled_fallback_budget_is_enforced_before_objects_are_written", "crates/edit/src/tests.rs::styled_fallback_same_base_font_and_character_keep_both_weights", "crates/automation/tests/it/automation.rs::styled_cyrillic_fallback_is_batched_through_tools_and_reopen", "crates/edit/src/tests.rs::an_edited_shaped_run_maps_each_glyph_to_its_own_characters", "crates/automation/tests/it/automation.rs::a_word_in_an_edited_shaped_line_is_found_on_its_own", "crates/fonts/src/source_shaping.rs::clusters_map_each_glyph_to_its_character_where_they_correspond", "crates/fonts/src/source_shaping.rs::real_thai_shaping_maps_glyphs_only_in_typing_order", "crates/fonts/src/source_shaping.rs::a_cluster_of_many_marks_looks_each_character_up_once", "crates/fonts/src/source_shaping.rs::a_thai_cluster_is_mapped_glyph_by_glyph_only_when_every_glyph_is_placed", "crates/fonts/src/source_shaping.rs::split_clusters_takes_empty_and_short_cluster_lists"] notes = "The Edit tab turns Edit text & images on at once, as in Acrobat (leaving the tab puts the tool down and applies an edit in progress); click a paragraph box and retype it in place (font reused or Helvetica substituted, colour and spacing kept, rewrapped); a page's /Contents pieces are parsed joined, so a line whose operands and operator sit in different pieces is listed and edited (lines and paragraphs still don't join across pieces; Edit image still parses each piece alone); drag a box to move it or its right-edge handle to rewrap it (edit.text-move-resize); per-character formatting changes are next Since #1116: mixed-style visual regions retain source rows, subset encodings, colour and baseline anchors in one textless-background editor. Explicit paragraph/heading scopes constrain grouping; compatible graphics planes permit cross-stream regions. Full matching document fonts complete absent subset glyphs; installed/file selection, TrueType/OpenType-CFF embedding and disjoint UTF-8 range styles have save/reopen coverage. Single-face OpenType shaping/bidi retains logical text through ToUnicode/ActualText. Replacement text the paragraph's font can't show takes the craft-fonts Japanese faces when they draw all of it (Uzbek қ, Kazakh ә are in none of them), else the bundled Inter face, so those letters write in every build (#2593). Legacy non-SFNT preview, mixed-font/style contextual runs, full structure-tree preservation and author intent in untagged PDFs remain limited; Styled-path outline fallback batches by actual face with 240 glyphs per font and preserves source styles; adjacent equal-font/style characters form runs. At most 15360 distinct fallback glyphs per styled paragraph; missing glyphs are refused before fallback objects are written. Glyph coverage, mixed contextual shaping, single-style fallback and stale-font cleanup are unchanged; see docs/editing-existing-text.md" [[feature]]