collective
Browse / github.com/storytold/designcraft / patch
Serbian hyphenation (Cyrillic and Latin) and installable hyph_*.dic pattern files
Serbian hyphenation (Cyrillic and Latin) and installable hyph_*.dic pattern files Serbian hyphenation in both scripts, and installable LibreOffice pattern files. Part of #1630 (the hyphenation half; spelling is not touched). ## What changes **Serbian, Cyrillic and Latin** - `hypher`'s `serbian` feature is on: `hyph-sh-cyrl.tex` 2.02 from hyph-utf8 (Dejan Muhamedagić, based on the Matica srpska *Pravopis*), LPPL 1 or later. It's listed in NOTICE, next to the other hypher pattern sets. - Those patterns are Cyrillic only. A Serbian word in Latin script is hyphenated through its Cyrillic spelling, letter for letter (`lj` → љ, `nj` → њ, `dž` → џ), and the breaks are mapped bac
- Status
- open
- Score
- 0.00
- Adopters
- 0 other operators
- Author
- sociolog (imported from pull request #1874, open)
- Base
ed8aabd1dc5a88652a076631c2d46886ca60c031- Size
- +492 −35 in 27 files
- Review
- pending
- Shared
- 2026-10-28
- Id
6fc9ce8ffd281c54384a683b
How it went
No other operator has reported using it yet.
Diff
raw · NOTICE README.md ROADMAP.md apps/designcraft/src/main.rs crates/compose/Cargo.toml crates/compose/src/hyphen/mod.rs crates/compose/src/hyphen/patgen.rs crates/compose/src/hyphen/patterns.rs crates/doc/src/attrs.rs crates/engine/src/hyphenation.rs crates/engine/src/lib.rs crates/engine/src/tests_file_guard.rs crates/ui-egui/src/i18n.rs crates/ui-egui/src/i18n/ar.rs crates/ui-egui/src/i18n/cs.rs crates/ui-egui/src/i18n/de.rs crates/ui-egui/src/i18n/it.rs crates/ui-egui/src/i18n/ja.rs crates/ui-egui/src/i18n/ka.rs crates/ui-egui/src/i18n/pl.rs crates/ui-egui/src/i18n/pt_br.rs crates/ui-egui/src/i18n/ru.rs crates/ui-egui/src/i18n/uk.rs crates/ui-egui/src/panels/properties.rs docs/gaps.md docs/localization-parity.md docs/typography-parity.md
@@ -74,6 +74,12 @@ list (author, source, license) is in ASSETS.md. with or without modification, is permitted in any medium without royalty provided the copyright notice and this notice are preserved (additionally distributed under the LaTeX Project Public License in CTAN distributions).+- Serbian hyphenation patterns, embedded through the hypher crate (feature "serbian"):+ hyph-sh-cyrl.tex 2.02 from hyph-utf8, https://github.com/hyphenation/tex-hyphen (patterns for+ Serbian in Cyrillic script; hypher maps the language code "sr" to these, and DesignCraft applies+ them to Serbian Latin text through its Cyrillic spelling).+ Copyright (C) 1990, 2008 Dejan Muhamedagić. Distributed under the LaTeX Project Public License,+ version 1 or (at your option) any later version, https://latex-project.org/lppl/. - Japanese fonts (craft-fonts): builds made with craft-fonts (CRAFT_FONTS_DIR; all official releases) embed the fonts listed in https://github.com/storytold/craft-fonts/blob/main/ATTRIBUTION.md (BIZ UDPGothic, BIZ UDMincho,@@ -160,6 +160,16 @@ The file stops growing at 16 MiB. `--version` writes no file. The logger is `apps/designcraft/src/logging.rs`; the web build logs to the browser console instead. +### Hyphenation patterns++LibreOffice/Hyphen pattern files (`hyph_<locale>.dic`, UTF-8, such as `hyph_sr_RS.dic` or+`hyph_de_DE.dic`) in a `dictionaries` folder of the settings directory, or in its subfolders+(one per locale, as InDesign's Hunspell folder keeps them: copy `sr_RS/` in as it is) (Linux+`~/.config/designcraft/dictionaries/`; macOS `~/Library/Application Support/DesignCraft/dictionaries/`;+Windows `%APPDATA%\DesignCraft\dictionaries\`) are read at start-up. Their patterns come before+the bundled ones for text in that language; words their alphabet doesn't cover (the other script)+still use the bundled patterns. The log lists each file read and how many patterns it gave.+ ### Portable mode A file named `portable.txt` next to the executable (on macOS, next to `DesignCraft.app`) makes the install portable: settings, logs and@@ -12,6 +12,11 @@ DesignCraft aims at full Adobe InDesign parity — and to be better: faster, open (documented JSON format + IDML), scriptable by agents (MCP), and available on the web. +- **2026-10-26** · Serbian hyphenation: Character › Language offers Serbian: Cyrillic and Serbian:+ Latin, both hyphenated by the hyph-sh-cyrl patterns (Latin text through its Cyrillic spelling, so+ lj, nj and dž stay whole). LibreOffice `hyph_<locale>.dic` pattern files in the settings folder's+ `dictionaries/` (or a locale subfolder of it) come before the bundled patterns for their language, and patterns may now be 16+ letters long instead of 9. No readiness percentage change. - **2026-10-25** · The application bar uses craft-window's shared title-bar code. A drag that starts on a floating panel over the bar moves the panel, not the window, and in full screen the bar no longer zooms or moves the window. No readiness percentage change.@@ -441,6 +441,13 @@ fn main() -> std::process::ExitCode { Err(e) => log::warn!("no log file: {e}"), } }+ // Hyphenation pattern files the user installed (`<settings>/dictionaries/hyph_*.dic`, also in+ // locale subfolders), before any text is composed.+ if let Some(dir) = profile::config_dir().filter(|_| !no_prefs()) {+ for line in designcraft_engine::hyphenation::install_dir(&dir.join("dictionaries")) {+ log::info!("{line}");+ }+ } let mut options = eframe::NativeOptions { viewport: egui::ViewportBuilder::default() .with_title("DesignCraft")@@ -21,7 +21,7 @@ log = { workspace = true } regex = { workspace = true } fancy-regex = { workspace = true } miniz_oxide = "0.8"-hypher = { version = "0.1.8", default-features = false, features = ["alloc", "greek", "portuguese", "spanish", "norwegian"] }+hypher = { version = "0.1.8", default-features = false, features = ["alloc", "greek", "portuguese", "spanish", "norwegian", "serbian"] } [dev-dependencies] designcraft-fonts = { workspace = true, features = ["testing"] }@@ -5,7 +5,14 @@ //! `hyph-no.tex` from hyph-utf8 (Rune Kleveland, Ole Michael Selberg), permissive (see NOTICE); //! Greek is `hyph-el-monoton.tex` 5.0 from hyph-utf8 (Dimitrios Filippou), LPPL or MIT, taken //! under MIT. Those are the **monotonic** patterns, which is what `hypher` maps `el` to, so-//! polytonic text hyphenates by them as well.+//! polytonic text hyphenates by them as well. Serbian is `hyph-sh-cyrl.tex` 2.02 from hyph-utf8+//! (Dejan Muhamedagić), LPPL 1 or later; the patterns are for Cyrillic, so Serbian words in Latin+//! script are hyphenated through their Cyrillic spelling ([`serbian_cyrillic`]).+//!+//! Pattern files can also be **installed** at run time ([`install_patterns`]): a+//! LibreOffice/Hyphen `hyph_xx_XX.dic` for a language comes before the bundled patterns, which+//! still cover the words its alphabet doesn't (a Latin-script set and a Cyrillic one can be+//! installed for the same language). //! //! English lookup order for each word: //! 1. the **dictionary** ([`Dictionary::en_us`]) — the public-domain Moby Hyphenator II word list@@ -21,7 +28,7 @@ pub mod dict; pub mod patgen; pub mod patterns; -use std::sync::OnceLock;+use std::sync::{OnceLock, PoisonError, RwLock}; pub use dict::Dictionary; pub use patterns::Patterns;@@ -73,26 +80,149 @@ impl Default for Limits { pub enum Lang { English, Patterns(hypher::Lang),+ /// Pattern files installed for a language ([`install_patterns`]), by index.+ Installed(u16), } impl Lang { /// The hyphenator for a language name or locale code (see [`designcraft_doc::language_tag`]).- /// None when the language has no bundled hyphenation.+ /// None when the language has neither installed nor bundled hyphenation. pub fn for_language(language: &str) -> Option<Lang> { if language.starts_with("English") { return Some(Lang::English); }- let tag = designcraft_doc::language_tag(language)?;- let sub = designcraft_doc::language_subtag(tag);+ let sub = pattern_language(language)?; if sub == "en" { return Some(Lang::English); }- let b = sub.as_bytes();- if b.len() != 2 {- return None;+ let installed = INSTALLED.read().unwrap_or_else(PoisonError::into_inner);+ if let Some(i) = installed.iter().position(|l| l.sub == sub) {+ return u16::try_from(i).ok().map(Lang::Installed);+ }+ bundled(sub).map(Lang::Patterns)+ }+}++/// The language subtag patterns are kept under: `sh` (Serbo-Croatian, the code Hunspell's Serbian+/// Latin files use) is Serbian.+fn pattern_language(language: &str) -> Option<&'static str> {+ let sub = designcraft_doc::language_subtag(designcraft_doc::language_tag(language)?);+ Some(if sub == "sh" { "sr" } else { sub })+}++/// The patterns compiled into `hypher` for a language subtag.+fn bundled(sub: &str) -> Option<hypher::Lang> {+ match *sub.as_bytes() {+ [a, b] => hypher::Lang::from_iso([a, b]),+ _ => None,+ }+}++/// Pattern sets installed for one language.+struct Installed {+ sub: &'static str,+ sets: Vec<Patterns>,+}++static INSTALLED: RwLock<Vec<Installed>> = RwLock::new(Vec::new());++/// At most this many installed pattern files per language, and languages.+const MAX_SETS: usize = 8;+const MAX_LANGUAGES: usize = 256;++/// Install a LibreOffice/Hyphen pattern file (see [`Patterns::parse_hyphen_dic`]) for `language`,+/// a locale code such as `sr_RS` (from the file name `hyph_sr_RS.dic`) or a language name. Its+/// patterns then come first for text in that language. Install before composing: hyphenated+/// words are cached. Returns how many patterns were read and how many skipped.+pub fn install_patterns(language: &str, text: &str) -> Result<(usize, usize), String> {+ let sub = pattern_language(language).ok_or_else(|| format!("{language}: not a language"))?;+ let (pats, skipped) = Patterns::parse_hyphen_dic(text).ok_or_else(|| format!("{language}: not a UTF-8 pattern file"))?;+ if pats.is_empty() {+ return Err(format!("{language}: no patterns"));+ }+ let n = pats.len();+ let mut installed = INSTALLED.write().unwrap_or_else(PoisonError::into_inner);+ let i = match installed.iter().position(|l| l.sub == sub) {+ Some(i) => i,+ None if installed.len() < MAX_LANGUAGES => {+ installed.push(Installed { sub, sets: Vec::new() });+ installed.len() - 1 }- hypher::Lang::from_iso([b[0], b[1]]).map(Lang::Patterns)+ None => return Err(format!("{language}: too many languages installed")),+ };+ let sets = &mut installed.get_mut(i).ok_or_else(|| format!("{language}: not installed"))?.sets;+ if sets.len() >= MAX_SETS {+ return Err(format!("{language}: too many pattern files"));+ }+ sets.push(pats);+ Ok((n, skipped))+}++/// Break points from the first pattern set installed under `i` whose alphabet covers `core`.+fn installed_points(i: u16, core: &[char]) -> Option<Vec<usize>> {+ let installed = INSTALLED.read().unwrap_or_else(PoisonError::into_inner);+ installed.get(usize::from(i))?.sets.iter().find_map(|p| p.points(core))+}++/// The bundled patterns behind installed language `i`.+fn installed_bundled(i: u16) -> Option<hypher::Lang> {+ let installed = INSTALLED.read().unwrap_or_else(PoisonError::into_inner);+ bundled(installed.get(usize::from(i))?.sub)+}++/// A lowercase Serbian Latin word in Cyrillic, letter for letter (`lj`, `nj`, `dž` are one+/// letter each: љ, њ, џ), with the Latin index each Cyrillic letter starts at. None when the word+/// has a letter outside the Serbian Latin alphabet. Words where such a pair is two letters+/// (`injekcija`, `nadživeti`) can't be told apart, so they break around the pair, not inside it.+fn serbian_cyrillic(word: &[char]) -> Option<(Vec<char>, Vec<usize>)> {+ let mut out = Vec::with_capacity(word.len());+ let mut at = Vec::with_capacity(word.len());+ let mut i = 0;+ while let Some(&c) = word.get(i) {+ let next = word.get(i + 1).copied();+ let (cyr, len) = match (c, next) {+ ('l', Some('j')) => ('љ', 2),+ ('n', Some('j')) => ('њ', 2),+ ('d', Some('ž')) => ('џ', 2),+ _ => (+ match c {+ 'a' => 'а',+ 'b' => 'б',+ 'c' => 'ц',+ 'č' => 'ч',+ 'ć' => 'ћ',+ 'd' => 'д',+ 'đ' => 'ђ',+ 'e' => 'е',+ 'f' => 'ф',+ 'g' => 'г',+ 'h' => 'х',+ 'i' => 'и',+ 'j' => 'ј',+ 'k' => 'к',+ 'l' => 'л',+ 'm' => 'м',+ 'n' => 'н',+ 'o' => 'о',+ 'p' => 'п',+ 'r' => 'р',+ 's' => 'с',+ 'š' => 'ш',+ 't' => 'т',+ 'u' => 'у',+ 'v' => 'в',+ 'z' => 'з',+ 'ž' => 'ж',+ _ => return None,+ },+ 1,+ ),+ };+ out.push(cyr);+ at.push(i);+ i += len; }+ Some((out, at)) } /// Where a word's break points came from.@@ -118,10 +248,28 @@ fn core_breaks(core: &[char], lang: Lang) -> (Vec<usize>, Source) { if n < 2 { return (vec![], Source::None); }+ let lang = match lang {+ Lang::Installed(i) => {+ if let Some(p) = installed_points(i, core) {+ return (p, Source::Patterns);+ }+ match installed_bundled(i) {+ Some(l) => Lang::Patterns(l),+ None => return (vec![], Source::None),+ }+ }+ _ => lang,+ }; if let Lang::Patterns(l) = lang { if core.contains(&'\'') { return (vec![], Source::None); }+ // Serbian Latin through its Cyrillic spelling, the script of the bundled patterns.+ if l == hypher::Lang::Serbian && core.iter().any(|c| c.is_ascii_alphabetic() || "čćđšž".contains(*c)) {+ let Some((cyr, at)) = serbian_cyrillic(core) else { return (vec![], Source::None) };+ let (pts, src) = core_breaks(&cyr, lang);+ return (pts.into_iter().filter_map(|p| at.get(p).copied()).collect(), src);+ } let s: String = core.iter().collect(); let mut out = Vec::new(); let mut at = 0;@@ -435,6 +583,67 @@ mod tests { assert_eq!(h("beveger"), "be-ve-ger"); } + #[test]+ fn serbian_uses_the_hyph_sh_patterns_in_both_scripts() {+ let sr = Lang::for_language("Serbian: Cyrillic").unwrap();+ for name in ["Serbian", "Serbian: Latin", "sr", "sr_RS", "sh_RS", "sr-Latn"] {+ assert_eq!(Lang::for_language(name), Some(sr), "{name}");+ }+ let h = |w: &str| {+ let pts = hyphen_points_in(w, &Limits::default(), sr);+ w.chars().enumerate().map(|(i, c)| if pts.contains(&i) { format!("-{c}") } else { c.to_string() }).collect::<String>()+ };+ assert_eq!(h("универзитет"), "уни-вер-зи-тет");+ assert_eq!(h("рачунар"), "ра-чу-нар");+ assert_eq!(h("Београду"), "Бео-гра-ду");+ // Latin hyphenates as its Cyrillic spelling, lj, nj and dž kept whole.+ assert_eq!(h("univerzitet"), "uni-ver-zi-tet");+ assert_eq!(h("računar"), "ra-ču-nar");+ assert_eq!(h("Beogradu"), "Beo-gra-du");+ assert_eq!(h("ljubavlju"), "lju-ba-vlju", "a consonant before a sonorant goes with it");+ assert_eq!(h("љубављу"), "љу-ба-вљу");+ assert_eq!(h("odgovornošću"), "od-go-vor-no-šću");+ // Both scripts break at the same letters.+ for (lat, cyr) in [("džungla", "џунгла"), ("njegovanje", "његовање"), ("đačkoj", "ђачкој"), ("Srbije", "Србије")]+ {+ let (l, c) = (h(lat), h(cyr));+ assert_eq!(+ serbian_cyrillic(&l.to_lowercase().replace('-', "").chars().collect::<Vec<_>>()).map(|x| x.0.iter().collect::<String>()),+ Some(cyr.to_lowercase())+ );+ assert_eq!(l.matches('-').count(), c.matches('-').count(), "{l} / {c}");+ }+ assert_eq!(h("xerox"), "xerox", "a letter outside the alphabet: no breaks");+ }++ #[test]+ fn serbian_latin_spells_in_cyrillic() {+ let w: Vec<char> = "džunglanja".chars().collect();+ let (cyr, at) = serbian_cyrillic(&w).unwrap();+ assert_eq!(cyr.iter().collect::<String>(), "џунглања");+ assert_eq!(at, vec![0, 2, 3, 4, 5, 6, 7, 9]);+ assert!(serbian_cyrillic(&['w']).is_none());+ }++ #[test]+ fn installed_patterns_come_first() {+ // A language no other test uses: installing is process-wide.+ assert!(Lang::for_language("tlh").is_none());+ assert!(install_patterns("tlh", "ISO8859-5\n1a\n").is_err());+ assert!(install_patterns("tlh", "UTF-8\nLEFTHYPHENMIN 2\n").is_err(), "no patterns");+ assert_eq!(install_patterns("tlh", "UTF-8\nLEFTHYPHENMIN 2\n1ba\n1ga\n").unwrap(), (2, 0));+ assert_eq!(install_patterns("tlh", "UTF-8\n1бa\n").unwrap(), (1, 0));+ let tlh = Lang::for_language("tlh").unwrap();+ assert!(matches!(tlh, Lang::Installed(_)));+ assert_eq!(hyphen_points_in("bagaga", &Limits::default(), tlh), vec![2, 4]);+ // Neither set's alphabet covers the word and nothing is bundled.+ assert!(hyphen_points_in("qqqqqq", &Limits::default(), tlh).is_empty());+ for _ in 0..MAX_SETS {+ let _ = install_patterns("tlh", "UTF-8\n1ba\n");+ }+ assert!(install_patterns("tlh", "UTF-8\n1ba\n").is_err(), "pattern files per language are capped");+ }+ #[test] fn heuristic_fallback() { let c: Vec<char> = "happen".chars().collect();@@ -9,6 +9,9 @@ use super::patterns::{Alphabet, FxMap, MAX_PATTERN_LEN, Patterns, apply, pack}; +/// Bit position of a candidate's dot (5 bits) in a counting key, above the packed letters.+const DOT_SHIFT: u32 = 6 * MAX_PATTERN_LEN as u32;+ /// Parameters of one level. #[derive(Clone, Copy, Debug)] pub struct Level {@@ -73,14 +76,14 @@ pub fn train(words: &[(Vec<char>, Vec<usize>)], levels: &[Level], log: &mut dyn let vals = vec![0; codes.len() + 1]; data.push(Word { codes, hyph, vals }); }- let mut pats: FxMap<u64, Box<[u8]>> = FxMap::default();+ let mut pats: FxMap<u128, Box<[u8]>> = FxMap::default(); let mut max_len = 0usize; for (li, lv) in levels.iter().enumerate() { let level = (li + 1) as u8; let hyphenating = level % 2 == 1; for len in lv.min_len..=lv.max_len.min(MAX_PATTERN_LEN) { // (packed letters | dot) → (good, bad)- let mut counts: FxMap<u64, (u32, u32)> = FxMap::default();+ let mut counts: FxMap<u128, (u32, u32)> = FxMap::default(); for w in &data { let n = w.codes.len(); // Positions strictly between the word's letters.@@ -102,7 +105,7 @@ pub fn train(words: &[(Vec<char>, Vec<usize>)], levels: &[Level], log: &mut dyn continue; } let s = p - dot;- let key = pack(&w.codes[s..s + len]) | ((dot as u64) << 54);+ let key = pack(&w.codes[s..s + len]) | ((dot as u128) << DOT_SHIFT); let e = counts.entry(key).or_default(); if good { e.0 += 1;@@ -113,12 +116,12 @@ pub fn train(words: &[(Vec<char>, Vec<usize>)], levels: &[Level], log: &mut dyn } } // Select.- let mut new: FxMap<u64, Box<[u8]>> = FxMap::default();+ let mut new: FxMap<u128, Box<[u8]>> = FxMap::default(); let mut selected = 0usize; for (key, (g, b)) in counts { if g > 0 && (g * lv.good_weight) as i64 - (b * lv.bad_weight) as i64 >= lv.threshold as i64 {- let dot = ((key >> 54) & 15) as usize;- let letters = key & !(15u64 << 54);+ let dot = ((key >> DOT_SHIFT) & 31) as usize;+ let letters = key & !(31u128 << DOT_SHIFT); let e = new.entry(letters).or_insert_with(|| vec![0u8; len + 1].into_boxed_slice()); e[dot] = level; selected += 1;@@ -9,10 +9,13 @@ use std::hash::{BuildHasherDefault, Hasher}; /// Word-boundary marker in patterns. pub const DOT: char = '.';-/// Longest pattern the packed key supports.-pub const MAX_PATTERN_LEN: usize = 9;+/// Longest pattern the packed key supports: 6 bits a letter below [`LEN_SHIFT`]. Installed pattern+/// files (hyph-utf8 and LibreOffice sets) have patterns of 10–11 letters, so 9 lost some.+pub const MAX_PATTERN_LEN: usize = 16;+/// Bit position of the letter count in a packed key.+pub const LEN_SHIFT: u32 = 120; -/// Small, fast multiplicative hasher for packed `u64` keys (FxHash-style).+/// Small, fast multiplicative hasher for packed keys (FxHash-style). #[derive(Default, Clone, Copy)] pub struct FxHasher(u64); @@ -28,6 +31,10 @@ impl Hasher for FxHasher { fn write_u64(&mut self, i: u64) { self.0 = (self.0.rotate_left(5) ^ i).wrapping_mul(0x51_7c_c1_b7_27_22_0a_95); }+ fn write_u128(&mut self, i: u128) {+ self.write_u64(i as u64);+ self.write_u64((i >> 64) as u64);+ } } pub type FxMap<K, V> = HashMap<K, V, BuildHasherDefault<FxHasher>>;@@ -74,12 +81,12 @@ impl Alphabet { /// Packed key of a code string (≤ [`MAX_PATTERN_LEN`] codes). #[inline]-pub fn pack(codes: &[u8]) -> u64 {- let mut k = 0u64;+pub fn pack(codes: &[u8]) -> u128 {+ let mut k = 0u128; for &c in codes {- k = (k << 6) | c as u64;+ k = (k << 6) | c as u128; }- k | ((codes.len() as u64) << 58)+ k | ((codes.len() as u128) << LEN_SHIFT) } /// A compiled pattern set.@@ -87,7 +94,7 @@ pub fn pack(codes: &[u8]) -> u64 { pub struct Patterns { alpha: Alphabet, /// Packed letters → inter-letter values (`len + 1` entries).- map: FxMap<u64, Box<[u8]>>,+ map: FxMap<u128, Box<[u8]>>, max_len: usize, } @@ -107,7 +114,38 @@ impl Patterns { p } - /// Add one pattern like `a1b`; returns false if it can't be represented.+ /// Parse a LibreOffice/Hyphen pattern file (`hyph_sr_RS.dic`): a charset line (only UTF-8 is+ /// read), then patterns as for [`Patterns::parse`]. The `LEFTHYPHENMIN`-style keyword lines+ /// are skipped (the paragraph's limits apply instead); of a two-level file (`NEXTLEVEL`) only+ /// the second level, the word patterns, is read; non-standard patterns (`/`) are skipped.+ /// Returns the patterns and how many were skipped, or None for another charset.+ pub fn parse_hyphen_dic(text: &str) -> Option<(Patterns, usize)> {+ let text = text.strip_prefix('\u{feff}').unwrap_or(text);+ let mut lines = text.lines();+ if !lines.next()?.trim().eq_ignore_ascii_case("UTF-8") {+ return None;+ }+ let body: Vec<&str> = lines.collect();+ let start = body.iter().rposition(|l| l.trim() == "NEXTLEVEL").map_or(0, |i| i + 1);+ let mut p = Patterns::default();+ let mut skipped = 0;+ for line in body.get(start..).unwrap_or_default() {+ let line = line.trim();+ if line.is_empty() || line.starts_with('%') || line.starts_with('#') || line.starts_with(|c: char| c.is_ascii_uppercase()) {+ continue;+ }+ for tok in line.split_whitespace() {+ if tok.contains('/') || !p.insert(&tok.to_lowercase()) {+ skipped += 1;+ }+ }+ }+ Some((p, skipped))+ }++ /// Add one pattern like `a1b`; returns false if it can't be represented. A pattern with the+ /// same letters as one already there (`b2j` and `bj2`, as pattern files have) is merged with+ /// it, the higher value at each position, as applying both would give. pub fn insert(&mut self, pat: &str) -> bool { let mut codes = Vec::with_capacity(pat.len()); let mut vals = vec![0u8];@@ -124,11 +162,14 @@ impl Patterns { return false; } self.max_len = self.max_len.max(codes.len());- self.map.insert(pack(&codes), vals.into_boxed_slice());+ let e = self.map.entry(pack(&codes)).or_insert_with(|| vec![0u8; vals.len()].into_boxed_slice());+ for (a, b) in e.iter_mut().zip(&vals) {+ *a = (*a).max(*b);+ } true } - pub(crate) fn from_parts(alpha: Alphabet, map: FxMap<u64, Box<[u8]>>, max_len: usize) -> Patterns {+ pub(crate) fn from_parts(alpha: Alphabet, map: FxMap<u128, Box<[u8]>>, max_len: usize) -> Patterns { Patterns { alpha, map, max_len } } @@ -182,7 +223,7 @@ impl Patterns { .map .iter() .map(|(&key, vals)| {- let n = (key >> 58) as usize;+ let n = (key >> LEN_SHIFT) as usize; let letters: Vec<char> = (0..n).map(|i| ch(((key >> (6 * (n - 1 - i))) & 63) as u8)).collect(); let mut s = String::new(); for (i, l) in letters.iter().enumerate() {@@ -204,13 +245,13 @@ impl Patterns { /// Apply `map` to padded `codes`, raising `vals` (len = codes.len() + 1). #[inline]-pub fn apply(map: &FxMap<u64, Box<[u8]>>, max_len: usize, codes: &[u8], vals: &mut [u8]) {+pub fn apply(map: &FxMap<u128, Box<[u8]>>, max_len: usize, codes: &[u8], vals: &mut [u8]) { let n = codes.len(); for s in 0..n {- let mut k = 0u64;+ let mut k = 0u128; for len in 1..=max_len.min(n - s) {- k = (k << 6) | codes[s + len - 1] as u64;- if let Some(v) = map.get(&(k | ((len as u64) << 58))) {+ k = (k << 6) | codes[s + len - 1] as u128;+ if let Some(v) = map.get(&(k | ((len as u128) << LEN_SHIFT))) { for (j, &x) in v.iter().enumerate() { if x > vals[s + j] { vals[s + j] = x;@@ -236,4 +277,38 @@ mod tests { assert!(tex.contains(&"hen5at".to_string())); assert!(p.values(&['h', 'é']).is_none()); }++ #[test]+ fn long_patterns_are_kept() {+ // 11 letters: over the old 9-letter limit, as in the Serbian pattern files.+ let mut p = Patterns::default();+ assert!(p.insert("1abcdefghi1jk"));+ let w: Vec<char> = "abcdefghijk".chars().collect();+ assert_eq!(p.points(&w).unwrap(), vec![9]);+ assert_eq!(p.to_tex(), vec!["1abcdefghi1jk".to_string()]);+ assert!(!p.insert(&"a".repeat(MAX_PATTERN_LEN + 1)));+ }++ #[test]+ fn same_letters_merge() {+ // Both sides of the pair: neither pattern may replace the other.+ let p = Patterns::parse("b2j bj2 1b");+ assert_eq!(p.to_tex(), vec!["1b".to_string(), "b2j2".to_string()]);+ let w: Vec<char> = "bjbj".chars().collect();+ assert_eq!(p.values(&w).unwrap(), vec![1, 2, 2, 2, 2]);+ }++ #[test]+ fn hyphen_dic_files() {+ let (p, skipped) = Patterns::parse_hyphen_dic("UTF-8\nLEFTHYPHENMIN 2\nRIGHTHYPHENMIN 2\n# comment\n1на\n1ти\nc1k/k=k,1,2\n").unwrap();+ assert_eq!(p.len(), 2, "the keyword lines are no patterns");+ assert_eq!(skipped, 1, "the non-standard pattern");+ let w: Vec<char> = "натина".chars().collect();+ assert_eq!(p.points(&w).unwrap(), vec![2, 4]);+ // Two levels: the second holds the word patterns.+ let (p, _) = Patterns::parse_hyphen_dic("UTF-8\n1x\nNEXTLEVEL\n1b\n").unwrap();+ assert_eq!(p.to_tex(), vec!["1b".to_string()]);+ assert!(Patterns::parse_hyphen_dic("ISO8859-5\n1a\n").is_none(), "only UTF-8 files");+ assert!(Patterns::parse_hyphen_dic("").is_none());+ } }@@ -305,6 +305,8 @@ pub fn language_tag(language: &str) -> Option<&'static str> { "portuguese brazilian" => Some("pt-BR"), "norwegian bokmål" | "norwegian bokmal" => Some("nb"), "norwegian nynorsk" => Some("nn"),+ "serbian cyrillic" | "sr cyrl" | "sr cyrl rs" => Some("sr-Cyrl"),+ "serbian latin" | "sr latn" | "sr latn rs" => Some("sr-Latn"), _ => None, }; if variant.is_some() {@@ -1627,6 +1629,11 @@ mod tests { ("English: USA", Some("en-US")), ("German: Swiss 2006 Reform", Some("de-CH")), ("Norwegian: Bokmål", Some("nb")),+ ("Serbian", Some("sr")),+ ("Serbian: Cyrillic", Some("sr-Cyrl")),+ ("Serbian: Latin", Some("sr-Latn")),+ ("sr-Latn", Some("sr-Latn")),+ ("sr_Cyrl_RS", Some("sr-Cyrl")), ("[No Language]", None), ] { assert_eq!(language_tag(name), tag, "{name}");new file mode 100644@@ -0,0 +1,106 @@+//! Hyphenation pattern files installed by the user: LibreOffice/Hyphen `hyph_<locale>.dic` files+//! (`hyph_sr_RS.dic`, `hyph_sh_RS.dic`, `hyph_de_DE.dic`) in a folder or its subfolders (one per+//! locale, as InDesign's Hunspell folder keeps them), read once at start-up. Their patterns come+//! before the bundled ones for text in that language+//! (see [`designcraft_compose::hyphen::install_patterns`]).++use std::path::{Path, PathBuf};++use designcraft_compose::hyphen;++/// Pattern files larger than this are skipped (the largest LibreOffice ones are under 1 MB).+const MAX_FILE_BYTES: u64 = 8 << 20;+/// At most this many files are read from the folder.+const MAX_FILES: usize = 64;++/// The locale of a pattern file name: `hyph_sr_RS.dic` → `sr_RS`.+fn locale_of(name: &str) -> Option<&str> {+ let locale = name.strip_prefix("hyph_")?.strip_suffix(".dic")?;+ (!locale.is_empty()).then_some(locale)+}++/// The files in `dir` and in its subfolders (one level), by name then path. A missing folder has+/// none. Names that aren't UTF-8 are left out.+pub(crate) fn dictionary_files(dir: &Path) -> Vec<(String, PathBuf)> {+ let list = |d: &Path| -> Vec<(String, PathBuf, bool)> {+ let Ok(entries) = std::fs::read_dir(d) else { return Vec::new() };+ entries+ .filter_map(|e| e.ok())+ .filter_map(|e| Some((e.file_name().into_string().ok()?, e.path(), e.file_type().is_ok_and(|t| t.is_dir()))))+ .collect()+ };+ let mut files = Vec::new();+ for (name, path, is_dir) in list(dir) {+ if is_dir {+ files.extend(list(&path).into_iter().filter(|f| !f.2).map(|(n, p, _)| (n, p)));+ } else {+ files.push((name, path));+ }+ }+ files.sort();+ files+}++/// Install the `hyph_*.dic` files in `dir` and its subfolders, in name order. A missing folder+/// installs nothing. Returns one line per file for the log: what was installed, or why not.+pub fn install_dir(dir: &Path) -> Vec<String> {+ let mut files: Vec<(String, PathBuf)> = dictionary_files(dir).into_iter().filter(|(name, _)| locale_of(name).is_some()).collect();+ let mut report = Vec::new();+ if files.len() > MAX_FILES {+ report.push(format!("hyphenation: only the first {MAX_FILES} of {} pattern files are read", files.len()));+ files.truncate(MAX_FILES);+ }+ for (name, path) in files {+ let Some(locale) = locale_of(&name) else { continue };+ let line = match std::fs::metadata(&path) {+ Ok(m) if !m.is_file() => continue,+ Ok(m) if m.len() > MAX_FILE_BYTES => format!("hyphenation: {name} skipped: larger than {} MB", MAX_FILE_BYTES >> 20),+ Err(e) => format!("hyphenation: {name}: {e}"),+ Ok(_) => match std::fs::read(&path) {+ Err(e) => format!("hyphenation: {name}: {e}"),+ Ok(bytes) => match String::from_utf8(bytes) {+ Err(_) => format!("hyphenation: {name} skipped: not UTF-8"),+ Ok(text) => match hyphen::install_patterns(locale, &text) {+ Ok((n, 0)) => format!("hyphenation: {name}: {n} patterns for {locale}"),+ Ok((n, skipped)) => format!("hyphenation: {name}: {n} patterns for {locale} ({skipped} not supported)"),+ Err(e) => format!("hyphenation: {name} skipped: {e}"),+ },+ },+ },+ };+ report.push(line);+ }+ report+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn pattern_file_names() {+ assert_eq!(locale_of("hyph_sr_RS.dic"), Some("sr_RS"));+ assert_eq!(locale_of("hyph_sh_RS.dic"), Some("sh_RS"));+ assert_eq!(locale_of("hyph_.dic"), None);+ assert_eq!(locale_of("sr_RS.dic"), None, "a spelling dictionary");+ assert_eq!(locale_of("hyph_sr_RS.dic.bak"), None);+ }++ #[test]+ fn installs_the_folder() {+ let dir = tempfile::tempdir().unwrap();+ // A language no other test installs or hyphenates: installing is process-wide.+ // A locale folder, as InDesign's Hunspell folder has them.+ std::fs::create_dir(dir.path().join("tpi")).unwrap();+ std::fs::write(dir.path().join("tpi/hyph_tpi.dic"), "UTF-8\nLEFTHYPHENMIN 2\n1pa\n1ta\n").unwrap();+ std::fs::write(dir.path().join("hyph_tpi_PG.dic"), "ISO8859-1\n1pa\n").unwrap();+ std::fs::write(dir.path().join("readme.txt"), "not a pattern file").unwrap();+ let report = install_dir(dir.path());+ assert_eq!(report.len(), 2, "{report:?}");+ assert!(report[0].contains("2 patterns for tpi"), "{report:?}");+ assert!(report[1].contains("skipped"), "{report:?}");+ let lang = hyphen::Lang::for_language("tpi").unwrap();+ assert_eq!(hyphen::hyphen_points_in("papatapa", &hyphen::Limits::default(), lang), vec![2, 4, 6]);+ assert!(install_dir(&dir.path().join("missing")).is_empty());+ }+}@@ -14,6 +14,7 @@ pub mod cmd; pub mod dtd; pub mod file_access; pub mod guard;+pub mod hyphenation; pub mod io; pub mod links; pub mod math;@@ -26,6 +26,10 @@ const ALLOWED: &[(&str, &str)] = &[ "engine/src/recovery.rs", "the unsaved documents live in a folder the host application chooses (a command's own `dir` is checked in cmd/file.rs before it gets here)", ),+ (+ "engine/src/hyphenation.rs",+ "reads the hyphenation pattern files in the dictionaries folder the host application chooses at start-up (`<settings>/dictionaries/`); no command parameter or document field names a path",+ ), ( "engine/src/cmd/package.rs", "`OutputRoot` checks the folder against the write root before it creates anything and then only writes inside it. Its reads are of the font files that draw the document's text (installed fonts and the `Document Fonts` folder), never of a path a command names",@@ -3780,6 +3780,8 @@ const TABLE: &[(&str, [&str; 5])] = &[ ("Portuguese: Brazilian", ["Portuguese: Brazilian", "Portugais : Brésil", "Portuguese: Brazilian", "Portuguese: Brazilian", "葡萄牙语：巴西"]), ("Rounded Rectangle", ["Rounded Rectangle", "Rectangle arrondi", "Rounded Rectangle", "Rounded Rectangle", "圆角矩形"]), ("Russian", ["Russian", "Russe", "Russian", "Russian", "俄语"]),+ ("Serbian: Cyrillic", ["Serbian: Cyrillic", "Serbe : cyrillique", "Serbian: Cyrillic", "Serbian: Cyrillic", "塞尔维亚语：西里尔字母"]),+ ("Serbian: Latin", ["Serbian: Latin", "Serbe : latin", "Serbian: Latin", "Serbian: Latin", "塞尔维亚语：拉丁字母"]), ("Solid", ["Solid", "Continu", "Solid", "Solid", "实线"]), ("Spanish", ["Spanish", "Espagnol", "Spanish", "Spanish", "西班牙语"]), ("Story Editor", ["Story Editor", "Mode éditeur", "Story Editor", "Story Editor", "文章编辑器"]),@@ -854,6 +854,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "البرتغالية: البرازيل"), ("Rounded Rectangle", "مستطيل مستدير الزوايا"), ("Russian", "الروسية"),+ ("Serbian: Cyrillic", "الصربية: السيريلية"),+ ("Serbian: Latin", "الصربية: اللاتينية"), ("Solid", "متصل"), ("Spanish", "الإسبانية"), ("Swedish", "السويدية"),@@ -1308,6 +1308,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portugalština: Brazílie"), ("Rounded Rectangle", "Zaoblený obdélník"), ("Russian", "Ruština"),+ ("Serbian: Cyrillic", "Srbština: cyrilice"),+ ("Serbian: Latin", "Srbština: latinka"), ("Solid", "Plná"), ("Spanish", "Španělština"), ("Story Editor", "Editor textu"),@@ -1377,6 +1377,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portugiesisch: Brasilien"), ("Rounded Rectangle", "Abgerundetes Rechteck"), ("Russian", "Russisch"),+ ("Serbian: Cyrillic", "Serbisch: Kyrillisch"),+ ("Serbian: Latin", "Serbisch: Lateinisch"), ("Solid", "Durchgezogen"), ("Spanish", "Spanisch"), ("Story Editor", "Textmodus"),@@ -1451,6 +1451,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portoghese: Brasile"), ("Rounded Rectangle", "Rettangolo arrotondato"), ("Russian", "Russo"),+ ("Serbian: Cyrillic", "Serbo: cirillico"),+ ("Serbian: Latin", "Serbo: latino"), ("Solid", "Continuo"), ("Spanish", "Spagnolo"), ("Story Editor", "Editor brani"),@@ -1552,6 +1552,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "ポルトガル語: ブラジル"), ("Rounded Rectangle", "角丸長方形"), ("Russian", "ロシア語"),+ ("Serbian: Cyrillic", "セルビア語: キリル文字"),+ ("Serbian: Latin", "セルビア語: ラテン文字"), ("Solid", "実線"), ("Spanish", "スペイン語"), ("Story Editor", "ストーリーエディター"),@@ -1407,6 +1407,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "პორტუგალიური: ბრაზილია"), ("Rounded Rectangle", "მომრგვალებული მართკუთხედი"), ("Russian", "რუსული"),+ ("Serbian: Cyrillic", "სერბული: კირილიცა"),+ ("Serbian: Latin", "სერბული: ლათინური"), ("Solid", "უწყვეტი"), ("Spanish", "ესპანური"), ("Story Editor", "ტექსტის რედაქტორი"),@@ -1340,6 +1340,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Portugalski: brazylijski"), ("Rounded Rectangle", "Zaokrąglony prostokąt"), ("Russian", "Rosyjski"),+ ("Serbian: Cyrillic", "Serbski: cyrylica"),+ ("Serbian: Latin", "Serbski: łacinka"), ("Solid", "Ciągła"), ("Spanish", "Hiszpański"), ("Story Editor", "Edytor wątków"),@@ -852,6 +852,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Português: Brasil"), ("Rounded Rectangle", "Retângulo arredondado"), ("Russian", "Russo"),+ ("Serbian: Cyrillic", "Sérvio: cirílico"),+ ("Serbian: Latin", "Sérvio: latino"), ("Solid", "Sólido"), ("Spanish", "Espanhol"), ("Swedish", "Sueco"),@@ -1296,6 +1296,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Португальский: Бразилия"), ("Rounded Rectangle", "Скругленный прямоугольник"), ("Russian", "Русский"),+ ("Serbian: Cyrillic", "Сербский: кириллица"),+ ("Serbian: Latin", "Сербский: латиница"), ("Solid", "Сплошной"), ("Spanish", "Испанский"), ("Story Editor", "Редактор материала"),@@ -1448,6 +1448,8 @@ pub(super) const TABLE: &[(&str, &str)] = &[ ("Portuguese: Brazilian", "Португальська: Бразилія"), ("Rounded Rectangle", "Округлений прямокутник"), ("Russian", "Російська"),+ ("Serbian: Cyrillic", "Сербська: кирилиця"),+ ("Serbian: Latin", "Сербська: латиниця"), ("Solid", "Суцільний"), ("Spanish", "Іспанська"), ("Story Editor", "Текстовий редактор"),@@ -1689,6 +1689,8 @@ pub(crate) const LANGUAGES: &[&str] = &[ "Greek", "Russian", "Ukrainian",+ "Serbian: Cyrillic",+ "Serbian: Latin", "Georgian", "Turkish", "Japanese",@@ -156,7 +156,7 @@ Kind: **F** feature · **U** UI/UX · **T** typography · **FF** file format · - **Estimate:** 25–40 h. ### 13. Hyphenation and spelling languages (T, L)-- **Missing:** hyphenation patterns for every language but English, Spanish, Portuguese, Norwegian and Greek; spelling for+- **Missing:** hyphenation patterns for every language but English, Spanish, Portuguese, Norwegian, Greek and Serbian (Cyrillic and Latin); spelling for every language but English. InDesign ships dictionaries for ~40 languages. - **Estimate:** 30–60 h (permissively licensed patterns exist for most; spelling word lists need licence review). **Doc:** [typography-parity.md](typography-parity.md)@@ -68,7 +68,7 @@ eight of the twelve (cs, da, el, en_GB, fi, hu, it, nb, nl, pl, ro, ru, sq, sv, ## Document languages InDesign 2026 hyphenates and spell-checks ~40 languages (Hunspell and Duden dictionaries).-DesignCraft hyphenates **English, Spanish, Portuguese and Norwegian** and spell-checks **English** only. Text language+DesignCraft hyphenates **English, Spanish, Portuguese, Norwegian, Greek and Serbian** (Cyrillic and Latin), plus any language whose LibreOffice `hyph_*.dic` pattern file is installed, and spell-checks **English** only. Text language can be set per paragraph and character (Properties, Character panel) and drives quotes and locale shaping. See [typography-parity.md](typography-parity.md). @@ -36,7 +36,7 @@ Fidelity: **unmeasured** unless stated. | Single-line Composer | greedy | greedy with the same ranges | ✓ | | | World-Ready composers | Paragraph / Single-line World-Ready | RTL, Arabic and bidi in the main composer | ~ | Indic scripts (Devanagari etc.) shape through harfrust but are untested | | Justification | word/letter spacing, glyph scaling min/desired/max, single-word, last line | all of these | ✓ | a justified line with a tab justifies after the tab (#284); set per paragraph in Type › Justification… and the Paragraph panel menu (#531) |-| Hyphenation | per-language dictionaries (~40), zone, limits, capitalized, last word, across columns | Liang patterns: English (trained from the public-domain Moby list), Spanish (Bezos), Portuguese (hyph-pt), Norwegian (hyph-no) and Greek (hyph-el-monoton); zone, limits, exceptions in the user dictionary; Hyphenate Across Column off keeps automatic hyphens off the last line of a column or frame | ~ | other languages don't hyphenate (#87 reported "doesn't work") |+| Hyphenation | per-language dictionaries (~40), zone, limits, capitalized, last word, across columns | Liang patterns: English (trained from the public-domain Moby list), Spanish (Bezos), Portuguese (hyph-pt), Norwegian (hyph-no), Greek (hyph-el-monoton) and Serbian (hyph-sh-cyrl, Latin text through its Cyrillic spelling); LibreOffice `hyph_*.dic` files installed in `<settings>/dictionaries/` (patterns up to 16 letters); zone, limits, exceptions in the user dictionary; Hyphenate Across Column off keeps automatic hyphens off the last line of a column or frame | ~ | other languages don't hyphenate (#87 reported "doesn't work") | | Spelling | per-language dictionaries | English (Moby) + document/user dictionary | ~ | | | Kerning | Metrics, Optical, manual, none | Metrics, Optical (outline-based: pairs pulled towards the font's stem gap, `crates/compose/src/optical.rs`), manual, none | ~ | Optical is an approximation of Adobe's algorithm (#181), not a match: imported display text can still break differently | | Tracking, scale, skew, baseline shift | ✓ | ✓ | ✓ | |