collective
Browse / github.com/storytold/designcraft / patch
Spelling in other languages with installed Hunspell dictionaries (Serbian Cyrillic and Latin tested)
Spelling in other languages with installed Hunspell dictionaries (Serbian Cyrillic and Latin tested) Spell checking in other languages with installed Hunspell dictionaries. This is the spelling half of #1630. **Stacked on #1874** (Serbian hyphenation), which adds the Serbian language entries and the `dictionaries/` folder this PR reads from. The first two commits here are #1874's; this PR's own change is the last commit. Once #1874 is merged, this diff shrinks to that commit. ## What changes - **Installed dictionaries.** At start-up, `<settings>/dictionaries/` and its locale subfolders are scanned for `<locale>.aff` + `<locale>.dic` pairs, laid out the way LibreOffice and InDesign's Hunspel
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- sociolog (imported from pull request #1875, open)
- Base
ed8aabd1dc5a88652a076631c2d46886ca60c031- Size
- +847 −59 in 32 files
- Review
- pending
- Shared
- 2026-10-28
- Id
40c180ccdd350f27c2a4fc23
How it went
No other operator has reported using it yet.
Diff
raw · Cargo.lock Cargo.toml NOTICE README.md ROADMAP.md apps/designcraft/src/main.rs crates/compose/Cargo.toml crates/compose/src/hyphen/mod.rs crates/compose/src/hyphen/patgen.rs crates/compose/src/hyphen/patterns.rs crates/doc/src/attrs.rs crates/engine/Cargo.toml crates/engine/src/cmd/spelling.rs crates/engine/src/dictionaries.rs crates/engine/src/hyphenation.rs crates/engine/src/lib.rs crates/engine/src/tests_file_guard.rs crates/ui-egui/src/i18n.rs crates/ui-egui/src/i18n/ar.rs crates/ui-egui/src/i18n/cs.rs crates/ui-egui/src/i18n/de.rs crates/ui-egui/src/i18n/it.rs crates/ui-egui/src/i18n/ja.rs crates/ui-egui/src/i18n/ka.rs crates/ui-egui/src/i18n/pl.rs crates/ui-egui/src/i18n/pt_br.rs crates/ui-egui/src/i18n/ru.rs crates/ui-egui/src/i18n/uk.rs crates/ui-egui/src/panels/properties.rs docs/gaps.md docs/localization-parity.md docs/typography-parity.md
@@ -1408,6 +1408,7 @@ dependencies = [ "same-file", "serde", "serde_json",+ "spellbook", "tempfile", "thiserror 2.0.21", "unicode-segmentation",@@ -5703,6 +5704,16 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]]+name = "spellbook"+version = "0.4.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0d204abcbdf8e88729306a8d0ca01a79d8a49969fe1f84696909d6dbc4321c1a"+dependencies = [+ "foldhash 0.2.0",+ "hashbrown 0.17.1",+]+ [[package]] name = "spirv" version = "0.4.0+sdk-1.4.341.0"@@ -53,6 +53,8 @@ unicode-script = "0.5" unicode-bidi-mirroring = "0.4" unicode-segmentation = "1" caseless = "0.2"+# Hunspell-compatible spell checking for installed dictionaries (MPL-2.0, Helix editor project).+spellbook = "0.4" regex = { version = "1", default-features = false, features = ["std", "unicode", "perf"] } fancy-regex = "0.19.2" tempfile = "3"@@ -74,6 +74,12 @@ list (author, source, license) is in ASSETS.md. with or without modification, is permitted in any medium without royalty provided the copyright notice and this notice are preserved (additionally distributed under the LaTeX Project Public License in CTAN distributions).+- Serbian hyphenation patterns, embedded through the hypher crate (feature "serbian"):+ hyph-sh-cyrl.tex 2.02 from hyph-utf8, https://github.com/hyphenation/tex-hyphen (patterns for+ Serbian in Cyrillic script; hypher maps the language code "sr" to these, and DesignCraft applies+ them to Serbian Latin text through its Cyrillic spelling).+ Copyright (C) 1990, 2008 Dejan Muhamedagić. Distributed under the LaTeX Project Public License,+ version 1 or (at your option) any later version, https://latex-project.org/lppl/. - Japanese fonts (craft-fonts): builds made with craft-fonts (CRAFT_FONTS_DIR; all official releases) embed the fonts listed in https://github.com/storytold/craft-fonts/blob/main/ATTRIBUTION.md (BIZ UDPGothic, BIZ UDMincho,@@ -87,6 +93,9 @@ list (author, source, license) is in ASSETS.md. Rust dependencies (vello_cpu, kurbo, skrifa, harfrust, image, egui, eframe, …) are permissively licensed (MIT/Apache-2.0/BSD/ISC/Zlib), fetched through crates.io or the pinned source below.+The spellbook crate (Hunspell-compatible spell checking for installed dictionaries,+https://github.com/helix-editor/spellbook) is under the Mozilla Public License 2.0, whose+source-file copyleft keeps it apart from DesignCraft's own licence; its source is on crates.io. No Adobe icons, images or artwork are included. @@ -160,6 +160,24 @@ The file stops growing at 16 MiB. `--version` writes no file. The logger is `apps/designcraft/src/logging.rs`; the web build logs to the browser console instead. +### Dictionaries++LibreOffice/Hyphen pattern files (`hyph_<locale>.dic`, UTF-8, such as `hyph_sr_RS.dic` or+`hyph_de_DE.dic`) in a `dictionaries` folder of the settings directory, or in its subfolders+(one per locale, as InDesign's Hunspell folder keeps them: copy `sr_RS/` in as it is) (Linux+`~/.config/designcraft/dictionaries/`; macOS `~/Library/Application Support/DesignCraft/dictionaries/`;+Windows `%APPDATA%\DesignCraft\dictionaries\`) are read at start-up. Their patterns come before+the bundled ones for text in that language; words their alphabet doesn't cover (the other script)+still use the bundled patterns. The log lists each file read and how many patterns it gave.++Hunspell spelling dictionaries (`<locale>.aff` beside `<locale>.dic`, UTF-8, such as+`sr_RS.aff` + `sr_RS.dic`) in the same folders spell-check text in their language (Edit ›+Spelling, Dynamic Spelling); English without one uses the bundled list. A dictionary is read the+first time text in its language is checked. With two for one language, as Serbian has for its+Cyrillic and Latin scripts, each word is checked by the one in its script. Check Spelling gives+suggestions from installed dictionaries for the first 10 misspelled words, as each takes a few+hundred milliseconds. Dictionaries are read by [spellbook](https://github.com/helix-editor/spellbook).+ ### Portable mode A file named `portable.txt` next to the executable (on macOS, next to `DesignCraft.app`) makes the install portable: settings, logs and@@ -12,6 +12,16 @@ DesignCraft aims at full Adobe InDesign parity — and to be better: faster, open (documented JSON format + IDML), scriptable by agents (MCP), and available on the web. +- **2026-10-26** · Spelling in other languages: Hunspell dictionaries (`<locale>.aff` + `.dic`) in+ the settings folder's `dictionaries/` (or a locale subfolder) spell-check text in their language,+ Check Spelling and Dynamic Spelling alike, read through spellbook on first use; with Serbian's+ Cyrillic and Latin dictionaries each word goes to the one in its script. No readiness+ percentage change.+- **2026-10-26** · Serbian hyphenation: Character › Language offers Serbian: Cyrillic and Serbian:+ Latin, both hyphenated by the hyph-sh-cyrl patterns (Latin text through its Cyrillic spelling, so+ lj, nj and dž stay whole). LibreOffice `hyph_<locale>.dic` pattern files in the settings folder's+ `dictionaries/` (or a locale subfolder of it) come before the bundled patterns for their language, and patterns may now be 16+ letters long instead of 9. No readiness percentage change. - **2026-10-25** · The application bar uses craft-window's shared title-bar code. A drag that starts on a floating panel over the bar moves the panel, not the window, and in full screen the bar no longer zooms or moves the window. No readiness percentage change.@@ -441,6 +441,15 @@ fn main() -> std::process::ExitCode { Err(e) => log::warn!("no log file: {e}"), } }+ // Dictionaries the user installed in `<settings>/dictionaries/` (also in locale subfolders):+ // hyphenation patterns (`hyph_*.dic`), read before any text is composed, and Hunspell spelling+ // dictionaries (`<locale>.aff` + `.dic`), read on first use.+ if let Some(dir) = profile::config_dir().filter(|_| !no_prefs()) {+ let dir = dir.join("dictionaries");+ for line in designcraft_engine::hyphenation::install_dir(&dir).into_iter().chain(designcraft_engine::dictionaries::install_dir(&dir)) {+ log::info!("{line}");+ }+ } let mut options = eframe::NativeOptions { viewport: egui::ViewportBuilder::default() .with_title("DesignCraft")@@ -21,7 +21,7 @@ log = { workspace = true } regex = { workspace = true } fancy-regex = { workspace = true } miniz_oxide = "0.8"-hypher = { version = "0.1.8", default-features = false, features = ["alloc", "greek", "portuguese", "spanish", "norwegian"] }+hypher = { version = "0.1.8", default-features = false, features = ["alloc", "greek", "portuguese", "spanish", "norwegian", "serbian"] } [dev-dependencies] designcraft-fonts = { workspace = true, features = ["testing"] }@@ -5,7 +5,14 @@ //! `hyph-no.tex` from hyph-utf8 (Rune Kleveland, Ole Michael Selberg), permissive (see NOTICE); //! Greek is `hyph-el-monoton.tex` 5.0 from hyph-utf8 (Dimitrios Filippou), LPPL or MIT, taken //! under MIT. Those are the **monotonic** patterns, which is what `hypher` maps `el` to, so-//! polytonic text hyphenates by them as well.+//! polytonic text hyphenates by them as well. Serbian is `hyph-sh-cyrl.tex` 2.02 from hyph-utf8+//! (Dejan Muhamedagić), LPPL 1 or later; the patterns are for Cyrillic, so Serbian words in Latin+//! script are hyphenated through their Cyrillic spelling ([`serbian_cyrillic`]).+//!+//! Pattern files can also be **installed** at run time ([`install_patterns`]): a+//! LibreOffice/Hyphen `hyph_xx_XX.dic` for a language comes before the bundled patterns, which+//! still cover the words its alphabet doesn't (a Latin-script set and a Cyrillic one can be+//! installed for the same language). //! //! English lookup order for each word: //! 1. the **dictionary** ([`Dictionary::en_us`]) — the public-domain Moby Hyphenator II word list@@ -21,7 +28,7 @@ pub mod dict; pub mod patgen; pub mod patterns; -use std::sync::OnceLock;+use std::sync::{OnceLock, PoisonError, RwLock}; pub use dict::Dictionary; pub use patterns::Patterns;@@ -73,26 +80,149 @@ impl Default for Limits { pub enum Lang { English, Patterns(hypher::Lang),+ /// Pattern files installed for a language ([`install_patterns`]), by index.+ Installed(u16), } impl Lang { /// The hyphenator for a language name or locale code (see [`designcraft_doc::language_tag`]).- /// None when the language has no bundled hyphenation.+ /// None when the language has neither installed nor bundled hyphenation. pub fn for_language(language: &str) -> Option<Lang> { if language.starts_with("English") { return Some(Lang::English); }- let tag = designcraft_doc::language_tag(language)?;- let sub = designcraft_doc::language_subtag(tag);+ let sub = pattern_language(language)?; if sub == "en" { return Some(Lang::English); }- let b = sub.as_bytes();- if b.len() != 2 {- return None;+ let installed = INSTALLED.read().unwrap_or_else(PoisonError::into_inner);+ if let Some(i) = installed.iter().position(|l| l.sub == sub) {+ return u16::try_from(i).ok().map(Lang::Installed);+ }+ bundled(sub).map(Lang::Patterns)+ }+}++/// The language subtag patterns are kept under: `sh` (Serbo-Croatian, the code Hunspell's Serbian+/// Latin files use) is Serbian.+fn pattern_language(language: &str) -> Option<&'static str> {+ let sub = designcraft_doc::language_subtag(designcraft_doc::language_tag(language)?);+ Some(if sub == "sh" { "sr" } else { sub })+}++/// The patterns compiled into `hypher` for a language subtag.+fn bundled(sub: &str) -> Option<hypher::Lang> {+ match *sub.as_bytes() {+ [a, b] => hypher::Lang::from_iso([a, b]),+ _ => None,+ }+}++/// Pattern sets installed for one language.+struct Installed {+ sub: &'static str,+ sets: Vec<Patterns>,+}++static INSTALLED: RwLock<Vec<Installed>> = RwLock::new(Vec::new());++/// At most this many installed pattern files per language, and languages.+const MAX_SETS: usize = 8;+const MAX_LANGUAGES: usize = 256;++/// Install a LibreOffice/Hyphen pattern file (see [`Patterns::parse_hyphen_dic`]) for `language`,+/// a locale code such as `sr_RS` (from the file name `hyph_sr_RS.dic`) or a language name. Its+/// patterns then come first for text in that language. Install before composing: hyphenated+/// words are cached. Returns how many patterns were read and how many skipped.+pub fn install_patterns(language: &str, text: &str) -> Result<(usize, usize), String> {+ let sub = pattern_language(language).ok_or_else(|| format!("{language}: not a language"))?;+ let (pats, skipped) = Patterns::parse_hyphen_dic(text).ok_or_else(|| format!("{language}: not a UTF-8 pattern file"))?;+ if pats.is_empty() {+ return Err(format!("{language}: no patterns"));+ }+ let n = pats.len();+ let mut installed = INSTALLED.write().unwrap_or_else(PoisonError::into_inner);+ let i = match installed.iter().position(|l| l.sub == sub) {+ Some(i) => i,+ None if installed.len() < MAX_LANGUAGES => {+ installed.push(Installed { sub, sets: Vec::new() });+ installed.len() - 1 }- hypher::Lang::from_iso([b[0], b[1]]).map(Lang::Patterns)+ None => return Err(format!("{language}: too many languages installed")),+ };+ let sets = &mut installed.get_mut(i).ok_or_else(|| format!("{language}: not installed"))?.sets;+ if sets.len() >= MAX_SETS {+ return Err(format!("{language}: too many pattern files"));+ }+ sets.push(pats);+ Ok((n, skipped))+}++/// Break points from the first pattern set installed under `i` whose alphabet covers `core`.+fn installed_points(i: u16, core: &[char]) -> Option<Vec<usize>> {+ let installed = INSTALLED.read().unwrap_or_else(PoisonError::into_inner);+ installed.get(usize::from(i))?.sets.iter().find_map(|p| p.points(core))+}++/// The bundled patterns behind installed language `i`.+fn installed_bundled(i: u16) -> Option<hypher::Lang> {+ let installed = INSTALLED.read().unwrap_or_else(PoisonError::into_inner);+ bundled(installed.get(usize::from(i))?.sub)+}++/// A lowercase Serbian Latin word in Cyrillic, letter for letter (`lj`, `nj`, `dž` are one+/// letter each: љ, њ, џ), with the Latin index each Cyrillic letter starts at. None when the word+/// has a letter outside the Serbian Latin alphabet. Words where such a pair is two letters+/// (`injekcija`, `nadživeti`) can't be told apart, so they break around the pair, not inside it.+fn serbian_cyrillic(word: &[char]) -> Option<(Vec<char>, Vec<usize>)> {+ let mut out = Vec::with_capacity(word.len());+ let mut at = Vec::with_capacity(word.len());+ let mut i = 0;+ while let Some(&c) = word.get(i) {+ let next = word.get(i + 1).copied();+ let (cyr, len) = match (c, next) {+ ('l', Some('j')) => ('љ', 2),+ ('n', Some('j')) => ('њ', 2),+ ('d', Some('ž')) => ('џ', 2),+ _ => (+ match c {+ 'a' => 'а',+ 'b' => 'б',+ 'c' => 'ц',+ 'č' => 'ч',+ 'ć' => 'ћ',+ 'd' => 'д',+ 'đ' => 'ђ',+ 'e' => 'е',+ 'f' => 'ф',+ 'g' => 'г',+ 'h' => 'х',+ 'i' => 'и',+ 'j' => 'ј',+ 'k' => 'к',+ 'l' => 'л',+ 'm' => 'м',+ 'n' => 'н',+ 'o' => 'о',+ 'p' => 'п',+ 'r' => 'р',+ 's' => 'с',+ 'š' => 'ш',+ 't' => 'т',+ 'u' => 'у',+ 'v' => 'в',+ 'z' => 'з',+ 'ž' => 'ж',+ _ => return None,+ },+ 1,+ ),+ };+ out.push(cyr);+ at.push(i);+ i += len; }+ Some((out, at)) } /// Where a word's break points came from.@@ -118,10 +248,28 @@ fn core_breaks(core: &[char], lang: Lang) -> (Vec<usize>, Source) { if n < 2 { return (vec![], Source::None); }+ let lang = match lang {+ Lang::Installed(i) => {+ if let Some(p) = installed_points(i, core) {+ return (p, Source::Patterns);+ }+ match installed_bundled(i) {+ Some(l) => Lang::Patterns(l),+ None => return (vec![], Source::None),+ }+ }+ _ => lang,+ }; if let Lang::Patterns(l) = lang { if core.contains(&'\'') { return (vec![], Source::None); }+ // Serbian Latin through its Cyrillic spelling, the script of the bundled patterns.+ if l == hypher::Lang::Serbian && core.iter().any(|c| c.is_ascii_alphabetic() || "čćđšž".contains(*c)) {+ let Some((cyr, at)) = serbian_cyrillic(core) else { return (vec![], Source::None) };+ let (pts, src) = core_breaks(&cyr, lang);+ return (pts.into_iter().filter_map(|p| at.get(p).copied()).collect(), src);+ } let s: String = core.iter().collect(); let mut out = Vec::new(); let mut at = 0;@@ -435,6 +583,67 @@ mod tests { assert_eq!(h("beveger"), "be-ve-ger"); } + #[test]+ fn serbian_uses_the_hyph_sh_patterns_in_both_scripts() {+ let sr = Lang::for_language("Serbian: Cyrillic").unwrap();+ for name in ["Serbian", "Serbian: Latin", "sr", "sr_RS", "sh_RS", "sr-Latn"] {+ assert_eq!(Lang::for_language(name), Some(sr), "{name}");+ }+ let h = |w: &str| {+ let pts = hyphen_points_in(w, &Limits::default(), sr);+ w.chars().enumerate().map(|(i, c)| if pts.contains(&i) { format!("-{c}") } else { c.to_string() }).collect::<String>()+ };+ assert_eq!(h("универзитет"), "уни-вер-зи-тет");+ assert_eq!(h("рачунар"), "ра-чу-нар");+ assert_eq!(h("Београду"), "Бео-гра-ду");+ // Latin hyphenates as its Cyrillic spelling, lj, nj and dž kept whole.+ assert_eq!(h("univerzitet"), "uni-ver-zi-tet");+ assert_eq!(h("računar"), "ra-ču-nar");+ assert_eq!(h("Beogradu"), "Beo-gra-du");+ assert_eq!(h("ljubavlju"), "lju-ba-vlju", "a consonant before a sonorant goes with it");+ assert_eq!(h("љубављу"), "љу-ба-вљу");+ assert_eq!(h("odgovornošću"), "od-go-vor-no-šću");+ // Both scripts break at the same letters.+ for (lat, cyr) in [("džungla", "џунгла"), ("njegovanje", "његовање"), ("đačkoj", "ђачкој"), ("Srbije", "Србије")]+ {+ let (l, c) = (h(lat), h(cyr));+ assert_eq!(+ serbian_cyrillic(&l.to_lowercase().replace('-', "").chars().collect::<Vec<_>>()).map(|x| x.0.iter().collect::<String>()),+ Some(cyr.to_lowercase())+ );+ assert_eq!(l.matches('-').count(), c.matches('-').count(), "{l} / {c}");+ }+ assert_eq!(h("xerox"), "xerox", "a letter outside the alphabet: no breaks");+ }++ #[test]+ fn serbian_latin_spells_in_cyrillic() {+ let w: Vec<char> = "džunglanja".chars().collect();+ let (cyr, at) = serbian_cyrillic(&w).unwrap();+ assert_eq!(cyr.iter().collect::<String>(), "џунглања");+ assert_eq!(at, vec![0, 2, 3, 4, 5, 6, 7, 9]);+ assert!(serbian_cyrillic(&['w']).is_none());+ }++ #[test]+ fn installed_patterns_come_first() {+ // A language no other test uses: installing is process-wide.+ assert!(Lang::for_language("tlh").is_none());+ assert!(install_patterns("tlh", "ISO8859-5\n1a\n").is_err());+ assert!(install_patterns("tlh", "UTF-8\nLEFTHYPHENMIN 2\n").is_err(), "no patterns");+ assert_eq!(install_patterns("tlh", "UTF-8\nLEFTHYPHENMIN 2\n1ba\n1ga\n").unwrap(), (2, 0));+ assert_eq!(install_patterns("tlh", "UTF-8\n1бa\n").unwrap(), (1, 0));+ let tlh = Lang::for_language("tlh").unwrap();+ assert!(matches!(tlh, Lang::Installed(_)));+ assert_eq!(hyphen_points_in("bagaga", &Limits::default(), tlh), vec![2, 4]);+ // Neither set's alphabet covers the word and nothing is bundled.+ assert!(hyphen_points_in("qqqqqq", &Limits::default(), tlh).is_empty());+ for _ in 0..MAX_SETS {+ let _ = install_patterns("tlh", "UTF-8\n1ba\n");+ }+ assert!(install_patterns("tlh", "UTF-8\n1ba\n").is_err(), "pattern files per language are capped");+ }+ #[test] fn heuristic_fallback() { let c: Vec<char> = "happen".chars().collect();@@ -9,6 +9,9 @@ use super::patterns::{Alphabet, FxMap, MAX_PATTERN_LEN, Patterns, apply, pack}; +/// Bit position of a candidate's dot (5 bits) in a counting key, above the packed letters.+const DOT_SHIFT: u32 = 6 * MAX_PATTERN_LEN as u32;+ /// Parameters of one level. #[derive(Clone, Copy, Debug)] pub struct Level {@@ -73,14 +76,14 @@ pub fn train(words: &[(Vec<char>, Vec<usize>)], levels: &[Level], log: &mut dyn let vals = vec![0; codes.len() + 1]; data.push(Word { codes, hyph, vals }); }- let mut pats: FxMap<u64, Box<[u8]>> = FxMap::default();+ let mut pats: FxMap<u128, Box<[u8]>> = FxMap::default(); let mut max_len = 0usize; for (li, lv) in levels.iter().enumerate() { let level = (li + 1) as u8; let hyphenating = level % 2 == 1; for len in lv.min_len..=lv.max_len.min(MAX_PATTERN_LEN) { // (packed letters | dot) → (good, bad)- let mut counts: FxMap<u64, (u32, u32)> = FxMap::default();+ let mut counts: FxMap<u128, (u32, u32)> = FxMap::default(); for w in &data { let n = w.codes.len(); // Positions strictly between the word's letters.@@ -102,7 +105,7 @@ pub fn train(words: &[(Vec<char>, Vec<usize>)], levels: &[Level], log: &mut dyn continue; } let s = p - dot;- let key = pack(&w.codes[s..s + len]) | ((dot as u64) << 54);+ let key = pack(&w.codes[s..s + len]) | ((dot as u128) << DOT_SHIFT); let e = counts.entry(key).or_default(); if good { e.0 += 1;@@ -113,12 +116,12 @@ pub fn train(words: &[(Vec<char>, Vec<usize>)], levels: &[Level], log: &mut dyn } } // Select.- let mut new: FxMap<u64, Box<[u8]>> = FxMap::default();+ let mut new: FxMap<u128, Box<[u8]>> = FxMap::default(); let mut selected = 0usize; for (key, (g, b)) in counts { if g > 0 && (g * lv.good_weight) as i64 - (b * lv.bad_weight) as i64 >= lv.threshold as i64 {- let dot = ((key >> 54) & 15) as usize;- let letters = key & !(15u64 << 54);+ let dot = ((key >> DOT_SHIFT) & 31) as usize;+ let letters = key & !(31u128 << DOT_SHIFT); let e = new.entry(letters).or_insert_with(|| vec![0u8; len + 1].into_boxed_slice()); e[dot] = level; selected += 1;@@ -9,10 +9,13 @@ use std::hash::{BuildHasherDefault, Hasher}; /// Word-boundary marker in patterns. pub const DOT: char = '.';-/// Longest pattern the packed key supports.-pub const MAX_PATTERN_LEN: usize = 9;+/// Longest pattern the packed key supports: 6 bits a letter below [`LEN_SHIFT`]. Installed pattern+/// files (hyph-utf8 and LibreOffice sets) have patterns of 10–11 letters, so 9 lost some.+pub const MAX_PATTERN_LEN: usize = 16;+/// Bit position of the letter count in a packed key.+pub const LEN_SHIFT: u32 = 120; -/// Small, fast multiplicative hasher for packed `u64` keys (FxHash-style).+/// Small, fast multiplicative hasher for packed keys (FxHash-style). #[derive(Default, Clone, Copy)] pub struct FxHasher(u64); @@ -28,6 +31,10 @@ impl Hasher for FxHasher { fn write_u64(&mut self, i: u64) { self.0 = (self.0.rotate_left(5) ^ i).wrapping_mul(0x51_7c_c1_b7_27_22_0a_95); }+ fn write_u128(&mut self, i: u128) {+ self.write_u64(i as u64);+ self.write_u64((i >> 64) as u64);+ } } pub type FxMap<K, V> = HashMap<K, V, BuildHasherDefault<FxHasher>>;@@ -74,12 +81,12 @@ impl Alphabet { /// Packed key of a code string (≤ [`MAX_PATTERN_LEN`] codes). #[inline]-pub fn pack(codes: &[u8]) -> u64 {- let mut k = 0u64;+pub fn pack(codes: &[u8]) -> u128 {+ let mut k = 0u128; for &c in codes {- k = (k << 6) | c as u64;+ k = (k << 6) | c as u128; }- k | ((codes.len() as u64) << 58)+ k | ((codes.len() as u128) << LEN_SHIFT) } /// A compiled pattern set.@@ -87,7 +94,7 @@ pub fn pack(codes: &[u8]) -> u64 { pub struct Patterns { alpha: Alphabet, /// Packed letters → inter-letter values (`len + 1` entries).- map: FxMap<u64, Box<[u8]>>,+ map: FxMap<u128, Box<[u8]>>, max_len: usize, } @@ -107,7 +114,38 @@ impl Patterns { p } - /// Add one pattern like `a1b`; returns false if it can't be represented.+ /// Parse a LibreOffice/Hyphen pattern file (`hyph_sr_RS.dic`): a charset line (only UTF-8 is+ /// read), then patterns as for [`Patterns::parse`]. The `LEFTHYPHENMIN`-style keyword lines+ /// are skipped (the paragraph's limits apply instead); of a two-level file (`NEXTLEVEL`) only+ /// the second level, the word patterns, is read; non-standard patterns (`/`) are skipped.+ /// Returns the patterns and how many were skipped, or None for another charset.+ pub fn parse_hyphen_dic(text: &str) -> Option<(Patterns, usize)> {+ let text = text.strip_prefix('\u{feff}').unwrap_or(text);+ let mut lines = text.lines();+ if !lines.next()?.trim().eq_ignore_ascii_case("UTF-8") {+ return None;+ }+ let body: Vec<&str> = lines.collect();+ let start = body.iter().rposition(|l| l.trim() == "NEXTLEVEL").map_or(0, |i| i + 1);+ let mut p = Patterns::default();+ let mut skipped = 0;+ for line in body.get(start..).unwrap_or_default() {+ let line = line.trim();+ if line.is_empty() || line.starts_with('%') || line.starts_with('#') || line.starts_with(|c: char| c.is_ascii_uppercase()) {+ continue;+ }+ for tok in line.split_whitespace() {+ if tok.contains('/') || !p.insert(&tok.to_lowercase()) {+ skipped += 1;+ }+ }+ }+ Some((p, skipped))+ }++ /// Add one pattern like `a1b`; returns false if it can't be represented. A pattern with the+ /// same letters as one already there (`b2j` and `bj2`, as pattern files have) is merged with+ /// it, the higher value at each position, as applying both would give. pub fn insert(&mut self, pat: &str) -> bool { let mut codes = Vec::with_capacity(pat.len()); let mut vals = vec![0u8];@@ -124,11 +162,14 @@ impl Patterns { return false; } self.max_len = self.max_len.max(codes.len());- self.map.insert(pack(&codes), vals.into_boxed_slice());+ let e = self.map.entry(pack(&codes)).or_insert_with(|| vec![0u8; vals.len()].into_boxed_slice());+ for (a, b) in e.iter_mut().zip(&vals) {+ *a = (*a).max(*b);+ } true } - pub(crate) fn from_parts(alpha: Alphabet, map: FxMap<u64, Box<[u8]>>, max_len: usize) -> Patterns {+ pub(crate) fn from_parts(alpha: Alphabet, map: FxMap<u128, Box<[u8]>>, max_len: usize) -> Patterns { Patterns { alpha, map, max_len } } @@ -182,7 +223,7 @@ impl Patterns { .map .iter() .map(|(&key, vals)| {- let n = (key >> 58) as usize;+ let n = (key >> LEN_SHIFT) as usize; let letters: Vec<char> = (0..n).map(|i| ch(((key >> (6 * (n - 1 - i))) & 63) as u8)).collect(); let mut s = String::new(); for (i, l) in letters.iter().enumerate() {@@ -204,13 +245,13 @@ impl Patterns { /// Apply `map` to padded `codes`, raising `vals` (len = codes.len() + 1). #[inline]-pub fn apply(map: &FxMap<u64, Box<[u8]>>, max_len: usize, codes: &[u8], vals: &mut [u8]) {+pub fn apply(map: &FxMap<u128, Box<[u8]>>, max_len: usize, codes: &[u8], vals: &mut [u8]) { let n = codes.len(); for s in 0..n {- let mut k = 0u64;+ let mut k = 0u128; for len in 1..=max_len.min(n - s) {- k = (k << 6) | codes[s + len - 1] as u64;- if let Some(v) = map.get(&(k | ((len as u64) << 58))) {+ k = (k << 6) | codes[s + len - 1] as u128;+ if let Some(v) = map.get(&(k | ((len as u128) << LEN_SHIFT))) { for (j, &x) in v.iter().enumerate() { if x > vals[s + j] { vals[s + j] = x;@@ -236,4 +277,38 @@ mod tests { assert!(tex.contains(&"hen5at".to_string())); assert!(p.values(&['h', 'é']).is_none()); }++ #[test]+ fn long_patterns_are_kept() {+ // 11 letters: over the old 9-letter limit, as in the Serbian pattern files.+ let mut p = Patterns::default();+ assert!(p.insert("1abcdefghi1jk"));+ let w: Vec<char> = "abcdefghijk".chars().collect();+ assert_eq!(p.points(&w).unwrap(), vec![9]);+ assert_eq!(p.to_tex(), vec!["1abcdefghi1jk".to_string()]);+ assert!(!p.insert(&"a".repeat(MAX_PATTERN_LEN + 1)));+ }++ #[test]+ fn same_letters_merge() {+ // Both sides of the pair: neither pattern may replace the other.+ let p = Patterns::parse("b2j bj2 1b");+ assert_eq!(p.to_tex(), vec!["1b".to_string(), "b2j2".to_string()]);+ let w: Vec<char> = "bjbj".chars().collect();+ assert_eq!(p.values(&w).unwrap(), vec![1, 2, 2, 2, 2]);+ }++ #[test]+ fn hyphen_dic_files() {+ let (p, skipped) = Patterns::parse_hyphen_dic("UTF-8\nLEFTHYPHENMIN 2\nRIGHTHYPHENMIN 2\n# comment\n1на\n1ти\nc1k/k=k,1,2\n").unwrap();+ assert_eq!(p.len(), 2, "the keyword lines are no patterns");+ assert_eq!(skipped, 1, "the non-standard pattern");+ let w: Vec<char> = "натина".chars().collect();+ assert_eq!(p.points(&w).unwrap(), vec![2, 4]);+ // Two levels: the second holds the word patterns.+ let (p, _) = Patterns::parse_hyphen_dic("UTF-8\n1x\nNEXTLEVEL\n1b\n").unwrap();+ assert_eq!(p.to_tex(), vec!["1b".to_string()]);+ assert!(Patterns::parse_hyphen_dic("ISO8859-5\n1a\n").is_none(), "only UTF-8 files");+ assert!(Patterns::parse_hyphen_dic("").is_none());+ } }@@ -305,6 +305,8 @@ pub fn language_tag(language: &str) -> Option<&'static str> { "portuguese brazilian" => Some("pt-BR"), "norwegian bokmål" | "norwegian bokmal" => Some("nb"), "norwegian nynorsk" => Some("nn"),+ "serbian cyrillic" | "sr cyrl" | "sr cyrl rs" => Some("sr-Cyrl"),+ "serbian latin" | "sr latn" | "sr latn rs" => Some("sr-Latn"), _ => None, }; if variant.is_some() {@@ -1627,6 +1629,11 @@ mod tests { ("English: USA", Some("en-US")), ("German: Swiss 2006 Reform", Some("de-CH")), ("Norwegian: Bokmål", Some("nb")),+ ("Serbian", Some("sr")),+ ("Serbian: Cyrillic", Some("sr-Cyrl")),+ ("Serbian: Latin", Some("sr-Latn")),+ ("sr-Latn", Some("sr-Latn")),+ ("sr_Cyrl_RS", Some("sr-Cyrl")), ("[No Language]", None), ] { assert_eq!(language_tag(name), tag, "{name}");@@ -34,6 +34,7 @@ log = { workspace = true } quick-xml = { workspace = true } caseless = { workspace = true } unicode-segmentation = { workspace = true }+spellbook = { workspace = true } [target.'cfg(not(target_arch = "wasm32"))'.dependencies] craft-atomic = { workspace = true }@@ -1,17 +1,21 @@-//! Edit → Spelling: check spelling against the bundled US English word list (the public-domain-//! Moby list used for hyphenation) plus the document's user dictionary; suggestions by edit distance.+//! Edit → Spelling: check English against the bundled US English word list (the public-domain+//! Moby list used for hyphenation), suggestions by edit distance; check a language with an+//! installed Hunspell dictionary against it ([`crate::dictionaries`]); the document's user+//! dictionary counts for both.++use std::sync::Arc; use designcraft_compose::hyphen::Dictionary; use designcraft_doc::StoryId; use serde_json::{Value, json}; use super::{CommandSpec, bad, cmd, has_doc, str_param};-use crate::{Result, Session};+use crate::{Result, Session, dictionaries}; pub fn specs() -> Vec<CommandSpec> { vec![ cmd!(query "spelling.check", "Check Spelling…", ["Edit", "Spelling"], Some("Cmd+I"),- "{story?, suggestions?: true} → [{story, start, end, word, suggestions}]", has_doc, check),+ "{story?, suggestions?: true} → [{story, start, end, word, suggestions}] (installed dictionaries suggest for the first 10 words)", has_doc, check), cmd!( "hyphenation.addException", "Add Hyphenation Exception",@@ -160,21 +164,66 @@ pub fn suggest(dict: &Dictionary, w: &str) -> Vec<String> { out } +/// What checks the words of a run of text.+#[derive(Clone)]+enum Checker {+ /// The bundled English list.+ English,+ /// Hunspell dictionaries installed for the run's language (see [`crate::dictionaries`]).+ Installed(Vec<Arc<dictionaries::Installed>>),+ /// Nothing: a language without a dictionary isn't checked.+ None,+}++impl Checker {+ fn for_language(language: &str) -> Checker {+ let installed = dictionaries::for_language(language);+ if !installed.is_empty() {+ Checker::Installed(installed)+ } else if designcraft_compose::is_english(language) {+ Checker::English+ } else {+ Checker::None+ }+ }++ /// Is `word` misspelled? The user dictionary counts.+ fn misspelled(&self, word: &str, user: &[String]) -> bool {+ match self {+ Checker::English => !known(Dictionary::en_us(), user, &word.to_lowercase()),+ Checker::Installed(dicts) => !user.iter().any(|u| *u == word.to_lowercase()) && dictionaries::check(dicts, word).is_some_and(|ok| !ok),+ Checker::None => false,+ }+ }+}+ /// Misspelled words of a story: byte ranges (acronyms skipped; the user dictionary counts). pub fn misspellings(doc: &designcraft_doc::Document, story: &designcraft_doc::Story, user: &[String]) -> Vec<std::ops::Range<usize>> {- // Text in other languages isn't checked against the English dictionary.- let foreign: Vec<std::ops::Range<usize>> = {- story- .runs()- .filter(|(r, f)| {- let pi = story.para_at(r.start.min(story.len()));- let base = doc.styles.resolve_para(&story.paras[pi.min(story.paras.len().saturating_sub(1))]).1;- !designcraft_compose::is_english(&doc.styles.resolve_char(&base, f).language)- })- .map(|(r, _)| r)- .collect()- };- let dict = Dictionary::en_us();+ misspellings_with(doc, story, user).into_iter().map(|(r, _)| r).collect()+}++/// [`misspellings`] with what found each one.+fn misspellings_with(doc: &designcraft_doc::Document, story: &designcraft_doc::Story, user: &[String]) -> Vec<(std::ops::Range<usize>, Checker)> {+ // Each run is checked in its language: English by the bundled list, a language with an+ // installed dictionary by that dictionary, any other not at all.+ let mut checkers: Vec<(String, Checker)> = Vec::new();+ let runs: Vec<(std::ops::Range<usize>, Checker)> = story+ .runs()+ .map(|(r, f)| {+ let pi = story.para_at(r.start.min(story.len()));+ let base = doc.styles.resolve_para(&story.paras[pi.min(story.paras.len().saturating_sub(1))]).1;+ let language = doc.styles.resolve_char(&base, f).language;+ let checker = match checkers.iter().find(|(l, _)| *l == language) {+ Some((_, c)) => c.clone(),+ None => {+ let c = Checker::for_language(&language);+ checkers.push((language, c.clone()));+ c+ }+ };+ (r, checker)+ })+ .collect(); let text = &story.text; let bytes: Vec<(usize, char)> = text.char_indices().collect(); let mut out = Vec::new();@@ -194,29 +243,44 @@ pub fn misspellings(doc: &designcraft_doc::Document, story: &designcraft_doc::St let end = bytes.get(j).map(|b| b.0).unwrap_or(text.len()); let word = &text[start..end]; let acronym = word.chars().all(|c| c.is_uppercase()) && word.chars().count() <= 5;- if !acronym && !foreign.iter().any(|r| r.contains(&start)) && !known(dict, user, &word.to_lowercase()) {- out.push(start..end);+ if !acronym+ && let Some((_, checker)) = runs.iter().find(|(r, _)| r.contains(&start))+ && checker.misspelled(word, user)+ {+ out.push((start..end, checker.clone())); } i = j.max(i + 1); } out } +/// Words that get suggestions from an installed dictionary in one check: those take a few+/// hundred milliseconds each (the English list's take microseconds, and all get them).+const DICTIONARY_SUGGESTIONS: usize = 10;+ fn check(s: &mut Session, p: &Value) -> Result<Value> { let st = s.doc()?; let d = &st.doc;- let dict = Dictionary::en_us(); let want_sugg = p.get("suggestions").and_then(Value::as_bool).unwrap_or(true); let stories: Vec<StoryId> = match p.get("story").and_then(Value::as_u64) { Some(id) => vec![StoryId(id)], None => d.stories.keys().copied().collect(), }; let mut out = Vec::new();+ let mut dictionary_suggestions = 0; for sid in stories { let Some(story) = d.story(sid) else { continue };- for r in misspellings(d, story, &d.user_words) {+ for (r, checker) in misspellings_with(d, story, &d.user_words) { let word = &story.text[r.clone()];- let sugg = if want_sugg { suggest(dict, &word.to_lowercase()) } else { vec![] };+ let sugg = match checker {+ _ if !want_sugg => vec![],+ Checker::English => suggest(Dictionary::en_us(), &word.to_lowercase()),+ Checker::Installed(dicts) if dictionary_suggestions < DICTIONARY_SUGGESTIONS => {+ dictionary_suggestions += 1;+ dictionaries::suggest(&dicts, word)+ }+ Checker::Installed(_) | Checker::None => vec![],+ }; out.push(json!({"story": sid.0, "start": r.start, "end": r.end, "word": word, "suggestions": sugg})); } }@@ -266,6 +330,28 @@ mod tests { s.execute("spelling.addWord", &json!({"word": "jumpd"})).unwrap(); assert_eq!(s.execute("spelling.check", &json!({})).unwrap().as_array().unwrap().len(), 1); }++ #[test]+ fn installed_dictionaries_check_their_language() {+ // A language no other test installs: installing is process-wide.+ let dir = tempfile::tempdir().unwrap();+ std::fs::write(dir.path().join("gd_GB.aff"), "SET UTF-8\nTRY abcdefghijklmnopqrstuvwxyz\n").unwrap();+ std::fs::write(dir.path().join("gd_GB.dic"), "2\nfailte\nslainte\n").unwrap();+ dictionaries::install_dir(dir.path());+ let mut s = Session::new();+ s.execute("file.new", &json!({})).unwrap();+ let r = s.execute("frame.create", &json!({"rect": [36, 36, 300, 200], "content": "text", "text": "failte slaintte Thw zzzq"})).unwrap();+ s.execute("text.select", &json!({"story": r["story"], "anchor": 0, "focus": 15})).unwrap();+ s.execute("type.char", &json!({"attrs": {"language": "gd"}})).unwrap();+ s.execute("text.select", &json!({"story": r["story"], "anchor": 20, "focus": 24})).unwrap();+ s.execute("type.char", &json!({"attrs": {"language": "Basque"}})).unwrap();+ let r = s.execute("spelling.check", &json!({})).unwrap();+ let words: Vec<&str> = r.as_array().unwrap().iter().map(|m| m["word"].as_str().unwrap()).collect();+ assert_eq!(words, vec!["slaintte", "Thw"], "Gaelic by its dictionary, English by the bundled list, Basque not at all");+ assert!(r[0]["suggestions"].as_array().unwrap().iter().any(|v| v == "slainte"));+ s.execute("spelling.addWord", &json!({"word": "Slaintte"})).unwrap();+ assert_eq!(s.execute("spelling.check", &json!({})).unwrap().as_array().unwrap().len(), 1, "the user dictionary counts");+ } } #[cfg(test)]new file mode 100644@@ -0,0 +1,208 @@+//! Hunspell spelling dictionaries installed by the user: `<locale>.aff` + `<locale>.dic` pairs+//! (`sr_RS`, `sh_RS`, `de_DE`, as LibreOffice and InDesign's Hunspell folder name them) in a+//! folder or its subfolders, registered at start-up and read on first use (a large dictionary+//! takes a few hundred milliseconds). Text in a language with an installed dictionary is+//! spell-checked with it, through [`spellbook`] (see `cmd::spelling`).++use std::path::{Path, PathBuf};+use std::sync::{Arc, OnceLock, PoisonError, RwLock};++/// Dictionary files larger than this are not read (the largest LibreOffice ones are ~20 MB).+const MAX_FILE_BYTES: u64 = 64 << 20;+/// At most this many dictionaries are registered.+const MAX_DICTIONARIES: usize = 32;+/// Letters looked at to tell a dictionary's script.+const SCRIPT_SAMPLE: usize = 4096;++/// One installed dictionary.+pub struct Installed {+ pub locale: String,+ language: &'static str,+ aff: PathBuf,+ dic: PathBuf,+ loaded: OnceLock<Option<Loaded>>,+}++struct Loaded {+ dict: spellbook::Dictionary,+ cyrillic: bool,+}++impl Installed {+ /// The dictionary, read on first use; None when it can't be read (logged once).+ fn loaded(&self) -> Option<&Loaded> {+ self.loaded+ .get_or_init(|| {+ let read = |p: &Path| -> Result<String, String> {+ let len = std::fs::metadata(p).map_err(|e| e.to_string())?.len();+ if len > MAX_FILE_BYTES {+ return Err(format!("larger than {} MB", MAX_FILE_BYTES >> 20));+ }+ String::from_utf8(std::fs::read(p).map_err(|e| e.to_string())?).map_err(|_| "not UTF-8".to_string())+ };+ let loaded = read(&self.aff).and_then(|aff| {+ let dic = read(&self.dic)?;+ let dict = spellbook::Dictionary::new(&aff, &dic).map_err(|e| e.to_string())?;+ Ok(Loaded { dict, cyrillic: mostly_cyrillic(&dic) })+ });+ match loaded {+ Ok(l) => {+ log::info!("spelling: {} dictionary read", self.locale);+ Some(l)+ }+ Err(e) => {+ log::warn!("spelling: {} dictionary skipped: {e}", self.locale);+ None+ }+ }+ })+ .as_ref()+ }+}++fn is_cyrillic(c: char) -> bool {+ matches!(c, '\u{0400}'..='\u{052F}')+}++/// Whether most of the first letters of a word list are Cyrillic.+fn mostly_cyrillic(dic: &str) -> bool {+ let (mut cyr, mut all) = (0usize, 0usize);+ for c in dic.chars().filter(|c| c.is_alphabetic()).take(SCRIPT_SAMPLE) {+ all += 1;+ cyr += usize::from(is_cyrillic(c));+ }+ cyr * 2 > all+}++static INSTALLED: RwLock<Vec<Arc<Installed>>> = RwLock::new(Vec::new());++/// The language dictionaries are kept under: `sh` (Serbo-Croatian, the code Hunspell's Serbian+/// Latin dictionaries use) is Serbian, as for hyphenation.+fn dictionary_language(language: &str) -> Option<&'static str> {+ let sub = designcraft_doc::language_subtag(designcraft_doc::language_tag(language)?);+ Some(if sub == "sh" { "sr" } else { sub })+}++/// Register the `<locale>.aff` + `<locale>.dic` pairs in `dir` and its subfolders (in name+/// order; `hyph_*.dic` hyphenation files are not dictionaries). A missing folder registers+/// nothing. Returns one line per dictionary for the log.+pub fn install_dir(dir: &Path) -> Vec<String> {+ let files = crate::hyphenation::dictionary_files(dir);+ let mut report = Vec::new();+ for (name, aff) in &files {+ let Some(locale) = name.strip_suffix(".aff").filter(|l| !l.is_empty()) else { continue };+ let dic = aff.with_extension("dic");+ if !dic.is_file() {+ report.push(format!("spelling: {name} skipped: no {locale}.dic beside it"));+ continue;+ }+ report.push(match install(locale, aff.clone(), dic) {+ Ok(()) => format!("spelling: {locale} dictionary installed"),+ Err(e) => format!("spelling: {name} skipped: {e}"),+ });+ }+ report+}++/// Register the dictionary `aff` + `dic` for `locale` (read on first use).+pub fn install(locale: &str, aff: PathBuf, dic: PathBuf) -> Result<(), String> {+ let language = dictionary_language(locale).ok_or_else(|| format!("{locale}: not a language"))?;+ let mut installed = INSTALLED.write().unwrap_or_else(PoisonError::into_inner);+ if installed.len() >= MAX_DICTIONARIES {+ return Err(format!("{locale}: too many dictionaries installed"));+ }+ installed.push(Arc::new(Installed { locale: locale.to_string(), language, aff, dic, loaded: OnceLock::new() }));+ Ok(())+}++/// The dictionaries installed for a language name or locale code; empty when there are none.+pub fn for_language(language: &str) -> Vec<Arc<Installed>> {+ let Some(language) = dictionary_language(language) else { return Vec::new() };+ INSTALLED.read().unwrap_or_else(PoisonError::into_inner).iter().filter(|d| d.language == language).cloned().collect()+}++/// The readable dictionaries of `dicts` in the script of `word` (all of them when none is).+fn for_word<'a>(dicts: &'a [Arc<Installed>], word: &str) -> Vec<&'a Loaded> {+ let loaded: Vec<&Loaded> = dicts.iter().filter_map(|d| d.loaded()).collect();+ let cyrillic = word.chars().find(|c| c.is_alphabetic()).is_some_and(is_cyrillic);+ let same: Vec<&Loaded> = loaded.iter().copied().filter(|l| l.cyrillic == cyrillic).collect();+ if same.is_empty() { loaded } else { same }+}++/// Is `word` (as written: dictionaries know which words are capitalized) spelled correctly by+/// one of `dicts`? None when none of them can be read.+pub fn check(dicts: &[Arc<Installed>], word: &str) -> Option<bool> {+ let loaded = for_word(dicts, word);+ (!loaded.is_empty()).then(|| loaded.iter().any(|l| l.dict.check(word)))+}++/// Up to 6 suggestions for `word` from the dictionaries in its script.+pub fn suggest(dicts: &[Arc<Installed>], word: &str) -> Vec<String> {+ let mut out: Vec<String> = Vec::new();+ for l in for_word(dicts, word) {+ let mut s = Vec::new();+ l.dict.suggest(word, &mut s);+ for w in s {+ if !out.contains(&w) {+ out.push(w);+ }+ }+ }+ out.truncate(6);+ out+}++#[cfg(test)]+mod tests {+ use super::*;++ // Small dictionaries in a language no other test installs: installing is process-wide.+ const AFF: &str = "SET UTF-8\nTRY abcdefghijklmnopqrstuvwxyz\nSFX A Y 1\nSFX A 0 s .\n";++ #[test]+ fn installs_and_checks_by_script() {+ let dir = tempfile::tempdir().unwrap();+ std::fs::create_dir(dir.path().join("kw_GB")).unwrap();+ std::fs::write(dir.path().join("kw_GB/kw_GB.aff"), AFF).unwrap();+ std::fs::write(dir.path().join("kw_GB/kw_GB.dic"), "3\nkernow/A\nlyver/A\nPenzans\n").unwrap();+ std::fs::write(dir.path().join("kw_GB/hyph_kw_GB.dic"), "UTF-8\n1a\n").unwrap();+ // The same language in Cyrillic, as Serbian has.+ std::fs::write(dir.path().join("kw.aff"), "SET UTF-8\n").unwrap();+ std::fs::write(dir.path().join("kw.dic"), "1\nкерноу\n").unwrap();+ std::fs::write(dir.path().join("kw_XX.aff"), AFF).unwrap();+ let report = install_dir(dir.path());+ assert_eq!(report.len(), 3, "{report:?}");+ assert!(report.iter().any(|l| l.contains("no kw_XX.dic")), "{report:?}");++ let kw = for_language("kw");+ assert_eq!(kw.len(), 2);+ assert_eq!(check(&kw, "kernow"), Some(true));+ assert_eq!(check(&kw, "kernows"), Some(true), "affixes apply");+ assert_eq!(check(&kw, "Penzans"), Some(true));+ assert_eq!(check(&kw, "penzans"), Some(false), "a capitalized word stays capitalized");+ assert_eq!(check(&kw, "керноу"), Some(true), "the Cyrillic dictionary");+ assert_eq!(check(&kw, "kernoq"), Some(false));+ assert!(suggest(&kw, "kernoq").contains(&"kernow".to_string()));+ assert!(suggest(&kw, "kernoq").iter().all(|s| !s.chars().any(is_cyrillic)), "suggestions in the word's script");+ assert!(for_language("Serbian").iter().all(|d| d.language == "sr"));+ assert!(for_language("[No Language]").is_empty());+ }++ #[test]+ fn unreadable_dictionaries_check_nothing() {+ let dir = tempfile::tempdir().unwrap();+ std::fs::write(dir.path().join("gv.aff"), AFF).unwrap();+ std::fs::write(dir.path().join("gv.dic"), [0xff, 0xfe, 0x00]).unwrap();+ assert_eq!(install_dir(dir.path()).len(), 1);+ assert_eq!(check(&for_language("gv"), "word"), None, "not UTF-8: read once, then skipped");+ }++ #[test]+ fn serbian_codes() {+ assert_eq!(dictionary_language("sr_RS"), Some("sr"));+ assert_eq!(dictionary_language("sh_RS"), Some("sr"));+ assert_eq!(dictionary_language("Serbian: Latin"), Some("sr"));+ assert!(mostly_cyrillic("2\nкерноу\nlyver\n"));+ assert!(!mostly_cyrillic("2\nkernow\nкер\n"));+ }+}new file mode 100644@@ -0,0 +1,106 @@+//! Hyphenation pattern files installed by the user: LibreOffice/Hyphen `hyph_<locale>.dic` files+//! (`hyph_sr_RS.dic`, `hyph_sh_RS.dic`, `hyph_de_DE.dic`) in a folder or its subfolders (one per+//! locale, as InDesign's Hunspell folder keeps them), read once at start-up. Their patterns come+//! before the bundled ones for text in that language+//! (see [`designcraft_compose::hyphen::install_patterns`]).++use std::path::{Path, PathBuf};++use designcraft_compose::hyphen;++/// Pattern files larger than this are skipped (the largest LibreOffice ones are under 1 MB).+const MAX_FILE_BYTES: u64 = 8 << 20;+/// At most this many files are read from the folder.+const MAX_FILES: usize = 64;++/// The locale of a pattern file name: `hyph_sr_RS.dic` → `sr_RS`.+fn locale_of(name: &str) -> Option<&str> {+ let locale = name.strip_prefix("hyph_")?.strip_suffix(".dic")?;+ (!locale.is_empty()).then_some(locale)+}++/// The files in `dir` and in its subfolders (one level), by name then path. A missing folder has+/// none. Names that aren't UTF-8 are left out.+pub(crate) fn dictionary_files(dir: &Path) -> Vec<(String, PathBuf)> {+ let list = |d: &Path| -> Vec<(String, PathBuf, bool)> {+ let Ok(entries) = std::fs::read_dir(d) else { return Vec::new() };+ entries+ .filter_map(|e| e.ok())+ .filter_map(|e| Some((e.file_name().into_string().ok()?, e.path(), e.file_type().is_ok_and(|t| t.is_dir()))))+ .collect()+ };+ let mut files = Vec::new();+ for (name, path, is_dir) in list(dir) {+ if is_dir {+ files.extend(list(&path).into_iter().filter(|f| !f.2).map(|(n, p, _)| (n, p)));+ } else {+ files.push((name, path));+ }+ }+ files.sort();+ files+}++/// Install the `hyph_*.dic` files in `dir` and its subfolders, in name order. A missing folder+/// installs nothing. Returns one line per file for the log: what was installed, or why not.+pub fn install_dir(dir: &Path) -> Vec<String> {+ let mut files: Vec<(String, PathBuf)> = dictionary_files(dir).into_iter().filter(|(name, _)| locale_of(name).is_some()).collect();+ let mut report = Vec::new();+ if files.len() > MAX_FILES {+ report.push(format!("hyphenation: only the first {MAX_FILES} of {} pattern files are read", files.len()));+ files.truncate(MAX_FILES);+ }+ for (name, path) in files {+ let Some(locale) = locale_of(&name) else { continue };+ let line = match std::fs::metadata(&path) {+ Ok(m) if !m.is_file() => continue,+ Ok(m) if m.len() > MAX_FILE_BYTES => format!("hyphenation: {name} skipped: larger than {} MB", MAX_FILE_BYTES >> 20),+ Err(e) => format!("hyphenation: {name}: {e}"),+ Ok(_) => match std::fs::read(&path) {+ Err(e) => format!("hyphenation: {name}: {e}"),+ Ok(bytes) => match String::from_utf8(bytes) {+ Err(_) => format!("hyphenation: {name} skipped: not UTF-8"),+ Ok(text) => match hyphen::install_patterns(locale, &text) {+ Ok((n, 0)) => format!("hyphenation: {name}: {n} patterns for {locale}"),+ Ok((n, skipped)) => format!("hyphenation: {name}: {n} patterns for {locale} ({skipped} not supported)"),+ Err(e) => format!("hyphenation: {name} skipped: {e}"),+ },+ },+ },+ };+ report.push(line);+ }+ report+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn pattern_file_names() {+ assert_eq!(locale_of("hyph_sr_RS.dic"), Some("sr_RS"));+ assert_eq!(locale_of("hyph_sh_RS.dic"), Some("sh_RS"));+ assert_eq!(locale_of("hyph_.dic"), None);+ assert_eq!(locale_of("sr_RS.dic"), None, "a spelling dictionary");+ assert_eq!(locale_of("hyph_sr_RS.dic.bak"), None);+ }++ #[test]+ fn installs_the_folder() {+ let dir = tempfile::tempdir().unwrap();+ // A language no other test installs or hyphenates: installing is process-wide.+ // A locale folder, as InDesign's Hunspell folder has them.+ std::fs::create_dir(dir.path().join("tpi")).unwrap();+ std::fs::write(dir.path().join("tpi/hyph_tpi.dic"), "UTF-8\nLEFTHYPHENMIN 2\n1pa\n1ta\n").unwrap();+ std::fs::write(dir.path().join("hyph_tpi_PG.dic"), "ISO8859-1\n1pa\n").unwrap();+ std::fs::write(dir.path().join("readme.txt"), "not a pattern file").unwrap();+ let report = install_dir(dir.path());+ assert_eq!(report.len(), 2, "{report:?}");+ assert!(report[0].contains("2 patterns for tpi"), "{report:?}");+ assert!(report[1].contains("skipped"), "{report:?}");+ let lang = hyphen::Lang::for_language("tpi").unwrap();+ assert_eq!(hyphen::hyphen_points_in("papatapa", &hyphen::Limits::default(), lang), vec![2, 4, 6]);+ assert!(install_dir(&dir.path().join("missing")).is_empty());+ }+}@@ -11,9 +11,11 @@ pub mod atomic; pub mod autosize; pub mod cmd;+pub mod dictionaries; pub mod dtd; pub mod file_access; pub mod guard;+pub mod hyphenation; pub mod io; pub mod links; pub mod math;@@ -26,6 +26,14 @@ const ALLOWED: &[(&str, &str)] = &[ "engine/src/recovery.rs", "the unsaved documents live in a folder the host application chooses (a command's own `dir` is checked in cmd/file.rs before it gets here)", ),+ (+ "engine/src/hyphenation.rs",+ "reads the hyphenation pattern files in the dictionaries folder the host application chooses at start-up (`<settings>/dictionaries/`); no command parameter or document field names a path",+ ),+ (+ "engine/src/dictionaries.rs",+ "reads the Hunspell dictionaries registered from the same host-chosen dictionaries folder; spelling commands name a language, never a path",+ ), ( "engine/src/cmd/package.rs", "`OutputRoot` checks the folder against the write root before it creates anything and then only writes inside it. Its reads are of the font files that draw the document's text (installed fonts and the `Document Fonts` folder), never of a path a command names",@@ -3780,6 +3780,8 @@ const TABLE: &[(&str, [&str; 5])] = &[ ("Portuguese: Brazilian", ["Portuguese: Brazilian", "Portugais : Brésil", "Portuguese: Brazilian", "Portuguese: Brazilian", "葡萄牙语：巴西"]), ("Rounded Rectangle", ["Rounded Rectangle", "Rectangle arrondi", "Rounded Rectangle", "Rounded Rectangle", "圆角矩形"]), ("Russian", ["Russian", "Russe", "Russian", "Russian", "俄语"]),+ ("Serbian: Cyrillic", ["Serbian: Cyrillic", "Serbe : cyrillique", "Serbian: Cyrillic", "Serbian: Cyrillic", "塞尔维亚语：西里尔字母"]),+ ("Serbian: Latin", ["Serbian: Latin", "Serbe : latin", "Serbian: Latin", "Serbian: Latin", "塞尔维亚语：拉丁字母"]), ("Solid", ["Solid", "Continu", "Solid", "Solid", "实线"]), ("Spanish", ["Spanish", "Espagnol", "Spanish", "Spanish", "西班牙语"]), ("Story Editor", ["Story Editor", "Mode éditeur", "Story Editor", "Story Editor", "文章编辑器"]),@@ -854,6 +854,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "البرتغالية: البرازيل"), ("Rounded Rectangle", "مستطيل مستدير الزوايا"), ("Russian", "الروسية"),+ ("Serbian: Cyrillic", "الصربية: السيريلية"),+ ("Serbian: Latin", "الصربية: اللاتينية"), ("Solid", "متصل"), ("Spanish", "الإسبانية"), ("Swedish", "السويدية"),@@ -1308,6 +1308,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portugalština: Brazílie"), ("Rounded Rectangle", "Zaoblený obdélník"), ("Russian", "Ruština"),+ ("Serbian: Cyrillic", "Srbština: cyrilice"),+ ("Serbian: Latin", "Srbština: latinka"), ("Solid", "Plná"), ("Spanish", "Španělština"), ("Story Editor", "Editor textu"),@@ -1377,6 +1377,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portugiesisch: Brasilien"), ("Rounded Rectangle", "Abgerundetes Rechteck"), ("Russian", "Russisch"),+ ("Serbian: Cyrillic", "Serbisch: Kyrillisch"),+ ("Serbian: Latin", "Serbisch: Lateinisch"), ("Solid", "Durchgezogen"), ("Spanish", "Spanisch"), ("Story Editor", "Textmodus"),@@ -1451,6 +1451,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portoghese: Brasile"), ("Rounded Rectangle", "Rettangolo arrotondato"), ("Russian", "Russo"),+ ("Serbian: Cyrillic", "Serbo: cirillico"),+ ("Serbian: Latin", "Serbo: latino"), ("Solid", "Continuo"), ("Spanish", "Spagnolo"), ("Story Editor", "Editor brani"),@@ -1552,6 +1552,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "ポルトガル語: ブラジル"), ("Rounded Rectangle", "角丸長方形"), ("Russian", "ロシア語"),+ ("Serbian: Cyrillic", "セルビア語: キリル文字"),+ ("Serbian: Latin", "セルビア語: ラテン文字"), ("Solid", "実線"), ("Spanish", "スペイン語"), ("Story Editor", "ストーリーエディター"),@@ -1407,6 +1407,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "პორტუგალიური: ბრაზილია"), ("Rounded Rectangle", "მომრგვალებული მართკუთხედი"), ("Russian", "რუსული"),+ ("Serbian: Cyrillic", "სერბული: კირილიცა"),+ ("Serbian: Latin", "სერბული: ლათინური"), ("Solid", "უწყვეტი"), ("Spanish", "ესპანური"), ("Story Editor", "ტექსტის რედაქტორი"),@@ -1340,6 +1340,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portugalski: brazylijski"), ("Rounded Rectangle", "Zaokrąglony prostokąt"), ("Russian", "Rosyjski"),+ ("Serbian: Cyrillic", "Serbski: cyrylica"),+ ("Serbian: Latin", "Serbski: łacinka"), ("Solid", "Ciągła"), ("Spanish", "Hiszpański"), ("Story Editor", "Edytor wątków"),@@ -852,6 +852,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Português: Brasil"), ("Rounded Rectangle", "Retângulo arredondado"), ("Russian", "Russo"),+ ("Serbian: Cyrillic", "Sérvio: cirílico"),+ ("Serbian: Latin", "Sérvio: latino"), ("Solid", "Sólido"), ("Spanish", "Espanhol"), ("Swedish", "Sueco"),@@ -1296,6 +1296,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Португальский: Бразилия"), ("Rounded Rectangle", "Скругленный прямоугольник"), ("Russian", "Русский"),+ ("Serbian: Cyrillic", "Сербский: кириллица"),+ ("Serbian: Latin", "Сербский: латиница"), ("Solid", "Сплошной"), ("Spanish", "Испанский"), ("Story Editor", "Редактор материала"),@@ -1448,6 +1448,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Португальська: Бразилія"), ("Rounded Rectangle", "Округлений прямокутник"), ("Russian", "Російська"),+ ("Serbian: Cyrillic", "Сербська: кирилиця"),+ ("Serbian: Latin", "Сербська: латиниця"), ("Solid", "Суцільний"), ("Spanish", "Іспанська"), ("Story Editor", "Текстовий редактор"),@@ -1689,6 +1689,8 @@ pub(crate) const LANGUAGES: &[&str] = &[ "Greek", "Russian", "Ukrainian",+ "Serbian: Cyrillic",+ "Serbian: Latin", "Georgian", "Turkish", "Japanese",@@ -156,8 +156,8 @@ Kind: **F** feature · **U** UI/UX · **T** typography · **FF** file format · - **Estimate:** 25–40 h. ### 13. Hyphenation and spelling languages (T, L)-- **Missing:** hyphenation patterns for every language but English, Spanish, Portuguese, Norwegian and Greek; spelling for- every language but English. InDesign ships dictionaries for ~40 languages.+- **Missing:** hyphenation patterns for every language but English, Spanish, Portuguese, Norwegian, Greek and Serbian (Cyrillic and Latin); spelling for+ every language but English without an installed Hunspell dictionary. InDesign ships dictionaries for ~40 languages. - **Estimate:** 30–60 h (permissively licensed patterns exist for most; spelling word lists need licence review). **Doc:** [typography-parity.md](typography-parity.md) @@ -68,7 +68,7 @@ eight of the twelve (cs, da, el, en_GB, fi, hu, it, nb, nl, pl, ro, ru, sq, sv, ## Document languages InDesign 2026 hyphenates and spell-checks ~40 languages (Hunspell and Duden dictionaries).-DesignCraft hyphenates **English, Spanish, Portuguese and Norwegian** and spell-checks **English** only. Text language+DesignCraft hyphenates **English, Spanish, Portuguese, Norwegian, Greek and Serbian** (Cyrillic and Latin), plus any language whose LibreOffice `hyph_*.dic` pattern file is installed, and spell-checks **English**, plus any language whose Hunspell dictionary is installed. Text language can be set per paragraph and character (Properties, Character panel) and drives quotes and locale shaping. See [typography-parity.md](typography-parity.md). @@ -36,8 +36,8 @@ Fidelity: **unmeasured** unless stated. | Single-line Composer | greedy | greedy with the same ranges | ✓ | | | World-Ready composers | Paragraph / Single-line World-Ready | RTL, Arabic and bidi in the main composer | ~ | Indic scripts (Devanagari etc.) shape through harfrust but are untested | | Justification | word/letter spacing, glyph scaling min/desired/max, single-word, last line | all of these | ✓ | a justified line with a tab justifies after the tab (#284); set per paragraph in Type › Justification… and the Paragraph panel menu (#531) |-| Hyphenation | per-language dictionaries (~40), zone, limits, capitalized, last word, across columns | Liang patterns: English (trained from the public-domain Moby list), Spanish (Bezos), Portuguese (hyph-pt), Norwegian (hyph-no) and Greek (hyph-el-monoton); zone, limits, exceptions in the user dictionary; Hyphenate Across Column off keeps automatic hyphens off the last line of a column or frame | ~ | other languages don't hyphenate (#87 reported "doesn't work") |-| Spelling | per-language dictionaries | English (Moby) + document/user dictionary | ~ | |+| Hyphenation | per-language dictionaries (~40), zone, limits, capitalized, last word, across columns | Liang patterns: English (trained from the public-domain Moby list), Spanish (Bezos), Portuguese (hyph-pt), Norwegian (hyph-no), Greek (hyph-el-monoton) and Serbian (hyph-sh-cyrl, Latin text through its Cyrillic spelling); LibreOffice `hyph_*.dic` files installed in `<settings>/dictionaries/` (patterns up to 16 letters); zone, limits, exceptions in the user dictionary; Hyphenate Across Column off keeps automatic hyphens off the last line of a column or frame | ~ | other languages don't hyphenate (#87 reported "doesn't work") |+| Spelling | per-language dictionaries | English (Moby) + document/user dictionary; installed Hunspell dictionaries (`<settings>/dictionaries/<locale>.aff`/`.dic`) for any language, through spellbook | ~ | | | Kerning | Metrics, Optical, manual, none | Metrics, Optical (outline-based: pairs pulled towards the font's stem gap, `crates/compose/src/optical.rs`), manual, none | ~ | Optical is an approximation of Adobe's algorithm (#181), not a match: imported display text can still break differently | | Tracking, scale, skew, baseline shift | ✓ | ✓ | ✓ | | | Character fill and stroke | fill, stroke, tint, weight, stroke aligned to centre or outside | fill, stroke, tints, weight, stroke behind fill (screen, PNG, PDF, IDML `StrokeAlignment`) | ~ | text strokes have no join or miter limit settings (#619 for stroke behind fill) |