One file per change for new UI strings and attribution rows (#6448)

New UI strings go in assets/locales/pending/<slug>.json and new ATTRIBUTION.md rows in
attribution/pending/<slug>.md. The app build, xtask lists/i18n/assets read them as part of the
catalogs, and the Generated files workflow folds and deletes them on main with
cargo xtask fold-pending.

Refs #6439

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bherbruck
2026-10-25 03:01:41 +00:00
committed by GitHub
parent 743accca7a
commit 1ceab2ed4c
18 changed files with 478 additions and 51 deletions
+9 -3
View File
@@ -1,6 +1,8 @@
# Files generated from the code are rewritten here, on main, after each merge, so pull requests
# never edit them (two pull requests that each regenerated one would always conflict):
# THIRD-PARTY-LICENSES.txt (`cargo xtask licences`) and docs/features.md (`cargo xtask features`).
# Pending files (assets/locales/pending, attribution/pending) are folded into the catalogs and
# ATTRIBUTION.md by `cargo xtask fold-pending`.
name: Generated files
on:
@@ -17,6 +19,8 @@ on:
- "crates/engine/**"
- "apps/solidcraft-cli/**"
- "xtask/**"
- "assets/locales/pending/*.json"
- "attribution/pending/*.md"
- ".github/workflows/generated.yml"
workflow_dispatch:
@@ -48,13 +52,15 @@ jobs:
run: cargo xtask licences
- name: docs/features.md
run: cargo xtask features
- name: Pending locale strings and attribution rows
run: cargo xtask fold-pending
- name: Commit if changed
run: |
if git diff --quiet -- THIRD-PARTY-LICENSES.txt docs/features.md; then echo "unchanged"; exit 0; fi
if [ -z "$(git status --porcelain)" ]; then echo "unchanged"; exit 0; fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add THIRD-PARTY-LICENSES.txt docs/features.md
git commit -m "Regenerate THIRD-PARTY-LICENSES.txt and docs/features.md [skip ci]"
git add -A THIRD-PARTY-LICENSES.txt docs/features.md assets/locales ATTRIBUTION.md attribution
git commit -m "Regenerate generated files and fold pending files [skip ci]"
for _ in 1 2 3; do
git pull --rebase origin main && git push origin HEAD:main && exit 0
sleep 10
+2
View File
@@ -218,3 +218,5 @@ workflow; never regenerate them in a PR.
Keep them conflict-free (CONTRIBUTING.md, "Lists many changes add to"): message catalogs sorted
(`cargo xtask lists --fix`), `mod` lines in their sorted group, the language list English first
then by id, generated files left to `main`, and a design-format change as one new file.
New UI strings go in a new `assets/locales/pending/<slug>.json` and new ATTRIBUTION.md rows in a new
`attribution/pending/<slug>.md`, never into the catalogs or the table (see the README in each folder).
+3
View File
@@ -280,6 +280,9 @@ and a fresh CI run:
after each merge by the Generated files workflow (`cargo xtask licences`, `cargo xtask
features`). Don't regenerate or edit them in a pull request.
- **Design format**: one file per change, never an edit to a shared number (above).
- **New UI strings and attribution rows**: one new file per change, `assets/locales/pending/<slug>.json`
and `attribution/pending/<slug>.md`, instead of editing the catalogs or the ATTRIBUTION.md table;
`cargo xtask fold-pending` folds them on `main` (see the README in each folder).
## Sending a change
+24
View File
@@ -0,0 +1,24 @@
# Pending UI strings
A pull request that adds UI strings adds **one new file here** instead of editing
`assets/locales/<locale>.json`. Two pull requests then never touch the same file, so they never
conflict.
- Name it after the change, lowercase with dashes: `sketch-trim-hints.json`.
- Contents: a map from locale code to `{"message id": "text"}`. Give English; add other locales
you can translate (a locale you leave out falls back to English, as before):
```json
{
"en": { "sketch.trim.hint": "Click a segment to trim it" },
"de": { "sketch.trim.hint": "Segment zum Kürzen anklicken" }
}
```
- The id must not already be in that locale's catalog, nor in another pending file; the build and
`cargo xtask lists` fail otherwise. To change an existing message, edit the catalog itself.
- Don't sort or fold anything. The app, `cargo xtask i18n` and the tests read these files as part of
the catalogs, so your strings work in your own pull request.
- After a merge, the Generated files workflow runs `cargo xtask fold-pending` on `main`: it adds the
messages to the catalogs (sorted, one per line) and deletes the files. Never run it in a pull
request.
+12
View File
@@ -0,0 +1,12 @@
# Pending attribution rows
A pull request that adds an asset adds **one new file here** instead of a row in `ATTRIBUTION.md`,
so two pull requests never edit the same table.
- Name it after the change, lowercase with dashes: `render-studio-example.md`.
- Contents: the rows only, in the format of the files table in `ATTRIBUTION.md`
(`| File | Author | Source | Licence |`); a header line is allowed and ignored.
- A file that already has a row in `ATTRIBUTION.md` or another pending file fails
`cargo xtask lists`. `cargo xtask assets` counts pending rows as attributed.
- After a merge, the Generated files workflow runs `cargo xtask fold-pending` on `main`: it appends
the rows to `ATTRIBUTION.md` and deletes the files. Never run it in a pull request.
+55
View File
@@ -36,12 +36,67 @@ fn main() {
}
println!("cargo:rerun-if-changed=build.rs");
scenario_tests();
if let Err(e) = merged_locales() {
eprintln!("locales: {e}");
std::process::exit(1);
}
if let Err(e) = icon_data() {
eprintln!("icon data: {e}");
std::process::exit(1);
}
}
/// `$OUT_DIR/locales/<id>.json`: each catalog in `assets/locales` plus the messages of
/// `assets/locales/pending/*.json` (`{"<locale>": {"<id>": "<text>"}}`, one file per change, folded
/// into the catalogs on main), so a pull request's new strings work before they are folded. An id
/// a catalog already has, or a locale without a catalog, fails the build, as `cargo xtask lists`
/// does.
fn merged_locales() -> Result<(), String> {
use serde_json::{Map, Value};
let dir = std::path::Path::new("../../assets/locales");
println!("cargo:rerun-if-changed={}", dir.display());
let mut catalogs: std::collections::BTreeMap<String, Map<String, Value>> = std::collections::BTreeMap::new();
for entry in std::fs::read_dir(dir).map_err(|e| format!("{}: {e}", dir.display()))?.filter_map(Result::ok) {
let path = entry.path();
if path.extension().is_none_or(|x| x != "json") {
continue;
}
let Some(id) = path.file_stem().and_then(|s| s.to_str()) else { continue };
let text = std::fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display()))?;
match serde_json::from_str(&text).map_err(|e| format!("{}: {e}", path.display()))? {
Value::Object(map) => catalogs.insert(id.to_string(), map),
_ => return Err(format!("{}: a catalog must be a JSON object", path.display())),
};
}
let pending = dir.join("pending");
println!("cargo:rerun-if-changed={}", pending.display());
let mut files: Vec<_> = std::fs::read_dir(&pending)
.map(|rd| rd.filter_map(|e| e.ok().map(|e| e.path())).filter(|p| p.extension().is_some_and(|x| x == "json")).collect())
.unwrap_or_default();
files.sort();
for file in files {
let text = std::fs::read_to_string(&file).map_err(|e| format!("{}: {e}", file.display()))?;
let map: std::collections::BTreeMap<String, std::collections::BTreeMap<String, String>> =
serde_json::from_str(&text).map_err(|e| format!("{}: expected {{\"<locale>\": {{\"<id>\": \"<text>\"}}}}: {e}", file.display()))?;
for (locale, messages) in map {
let catalog =
catalogs.get_mut(&locale).ok_or_else(|| format!("{}: `{locale}` has no catalog assets/locales/{locale}.json", file.display()))?;
for (id, text) in messages {
if catalog.insert(id.clone(), Value::String(text)).is_some() {
return Err(format!("{}: `{id}` is already in {locale}.json (or another pending file)", file.display()));
}
}
}
}
let out = std::path::Path::new(&std::env::var("OUT_DIR").map_err(|e| e.to_string())?).join("locales");
std::fs::create_dir_all(&out).map_err(|e| e.to_string())?;
for (id, catalog) in catalogs {
let text = serde_json::to_string(&catalog).map_err(|e| e.to_string())?;
std::fs::write(out.join(format!("{id}.json")), text).map_err(|e| e.to_string())?;
}
Ok(())
}
/// `$OUT_DIR/scenario_tests.rs`: a `#[test]` per scenario file, named after it; a scenario with
/// a `{"pending": "why"}` step is `#[ignore]`d with that reason (it waits for a fix).
fn scenario_tests() {
+9 -9
View File
@@ -879,7 +879,7 @@ fn the_shader_gets_the_modes_colours_and_its_stripes() {
#[test]
fn the_sections_messages_are_in_every_catalog() {
let load = |text: &str| -> serde_json::Map<String, serde_json::Value> { serde_json::from_str(text).unwrap() };
let english = load(include_str!("../../../assets/locales/en.json"));
let english = load(include_str!(concat!(env!("OUT_DIR"), "/locales/en.json")));
let ids: Vec<&str> = ColorVision::ALL
.into_iter()
.flat_map(|m| [m.label_id(), m.help_id()])
@@ -887,14 +887,14 @@ fn the_sections_messages_are_in_every_catalog() {
.chain(["ui.colour_vision", "ui.colour_vision_mode", "ui.colour_vision_sample", "ui.colour_vision_note"])
.collect();
for (lang, text) in [
("cs", include_str!("../../../assets/locales/cs.json")),
("da", include_str!("../../../assets/locales/da.json")),
("de", include_str!("../../../assets/locales/de.json")),
("ja", include_str!("../../../assets/locales/ja.json")),
("ko", include_str!("../../../assets/locales/ko.json")),
("pl", include_str!("../../../assets/locales/pl.json")),
("ru", include_str!("../../../assets/locales/ru.json")),
("zh", include_str!("../../../assets/locales/zh.json")),
("cs", include_str!(concat!(env!("OUT_DIR"), "/locales/cs.json"))),
("da", include_str!(concat!(env!("OUT_DIR"), "/locales/da.json"))),
("de", include_str!(concat!(env!("OUT_DIR"), "/locales/de.json"))),
("ja", include_str!(concat!(env!("OUT_DIR"), "/locales/ja.json"))),
("ko", include_str!(concat!(env!("OUT_DIR"), "/locales/ko.json"))),
("pl", include_str!(concat!(env!("OUT_DIR"), "/locales/pl.json"))),
("ru", include_str!(concat!(env!("OUT_DIR"), "/locales/ru.json"))),
("zh", include_str!(concat!(env!("OUT_DIR"), "/locales/zh.json"))),
] {
let catalog = load(text);
for id in &ids {
+3 -3
View File
@@ -81,7 +81,7 @@ mod tests {
#[test]
fn every_help_message_is_for_a_documented_command() {
let english = help_keys(include_str!("../../../assets/locales/en.json"));
let english = help_keys(include_str!(concat!(env!("OUT_DIR"), "/locales/en.json")));
let wanted: BTreeSet<String> = DOCUMENTED.iter().map(|id| format!("help.{id}")).collect();
assert_eq!(english, wanted);
for key in &english {
@@ -92,8 +92,8 @@ mod tests {
#[test]
fn the_help_texts_are_written_in_every_complete_language() {
let english = catalog(include_str!("../../../assets/locales/en.json"));
for source in [include_str!("../../../assets/locales/ja.json"), include_str!("../../../assets/locales/zh.json")] {
let english = catalog(include_str!(concat!(env!("OUT_DIR"), "/locales/en.json")));
for source in [include_str!(concat!(env!("OUT_DIR"), "/locales/ja.json")), include_str!(concat!(env!("OUT_DIR"), "/locales/zh.json"))] {
let other = catalog(source);
for (key, text) in english.iter().filter(|(k, _)| k.starts_with("help.")) {
let translated = other.get(key).unwrap_or_else(|| panic!("{key} missing"));
+39 -29
View File
@@ -31,36 +31,32 @@ impl Catalog {
}
}
static ENGLISH: Catalog = Catalog::new("en", "English", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/en.json")));
static CZECH: Catalog = Catalog::new("cs", "Čeština", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/cs.json")));
static DANISH: Catalog = Catalog::new("da", "Dansk", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/da.json")));
static GERMAN: Catalog = Catalog::new("de", "Deutsch", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/de.json")));
static ENGLISH: Catalog = Catalog::new("en", "English", include_str!(concat!(env!("OUT_DIR"), "/locales/en.json")));
static CZECH: Catalog = Catalog::new("cs", "Čeština", include_str!(concat!(env!("OUT_DIR"), "/locales/cs.json")));
static DANISH: Catalog = Catalog::new("da", "Dansk", include_str!(concat!(env!("OUT_DIR"), "/locales/da.json")));
static GERMAN: Catalog = Catalog::new("de", "Deutsch", include_str!(concat!(env!("OUT_DIR"), "/locales/de.json")));
/// English in UK spelling: only the messages whose spelling differs; the rest are English's.
static ENGLISH_UK: Catalog =
Catalog::new("en-GB", "English (UK)", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/en-GB.json")));
static SPANISH: Catalog = Catalog::new("es", "Español (España)", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/es.json")));
static FRENCH: Catalog = Catalog::new("fr", "Français", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/fr.json")));
static GALICIAN: Catalog = Catalog::new("gl", "Galego", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/gl.json")));
static HUNGARIAN: Catalog = Catalog::new("hu", "Magyar", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/hu.json")));
static INDONESIAN: Catalog =
Catalog::new("id", "Bahasa Indonesia", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/id.json")));
static ITALIAN: Catalog = Catalog::new("it", "Italiano", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/it.json")));
static JAPANESE: Catalog = Catalog::new("ja", "日本語", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/ja.json")));
static KOREAN: Catalog = Catalog::new("ko", "한국어", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/ko.json")));
static MALAY: Catalog = Catalog::new("ms", "Bahasa Melayu", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/ms.json")));
static DUTCH: Catalog = Catalog::new("nl", "Nederlands", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/nl.json")));
static POLISH: Catalog = Catalog::new("pl", "Polski", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/pl.json")));
static PORTUGUESE_BR: Catalog =
Catalog::new("pt-BR", "Português (Brasil)", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/pt-BR.json")));
static ROMANIAN: Catalog = Catalog::new("ro", "Română", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/ro.json")));
static RUSSIAN: Catalog = Catalog::new("ru", "Русский", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/ru.json")));
static SWEDISH: Catalog = Catalog::new("sv", "Svenska", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/sv.json")));
static TURKISH: Catalog = Catalog::new("tr", "Türkçe", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/tr.json")));
static UKRAINIAN: Catalog = Catalog::new("uk", "Українська", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/uk.json")));
static VIETNAMESE: Catalog = Catalog::new("vi", "Tiếng Việt", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/vi.json")));
static CHINESE: Catalog = Catalog::new("zh", "中文", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/zh.json")));
static CHINESE_TRADITIONAL: Catalog =
Catalog::new("zh-TW", "繁體中文", include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/zh-TW.json")));
static ENGLISH_UK: Catalog = Catalog::new("en-GB", "English (UK)", include_str!(concat!(env!("OUT_DIR"), "/locales/en-GB.json")));
static SPANISH: Catalog = Catalog::new("es", "Español (España)", include_str!(concat!(env!("OUT_DIR"), "/locales/es.json")));
static FRENCH: Catalog = Catalog::new("fr", "Français", include_str!(concat!(env!("OUT_DIR"), "/locales/fr.json")));
static GALICIAN: Catalog = Catalog::new("gl", "Galego", include_str!(concat!(env!("OUT_DIR"), "/locales/gl.json")));
static HUNGARIAN: Catalog = Catalog::new("hu", "Magyar", include_str!(concat!(env!("OUT_DIR"), "/locales/hu.json")));
static INDONESIAN: Catalog = Catalog::new("id", "Bahasa Indonesia", include_str!(concat!(env!("OUT_DIR"), "/locales/id.json")));
static ITALIAN: Catalog = Catalog::new("it", "Italiano", include_str!(concat!(env!("OUT_DIR"), "/locales/it.json")));
static JAPANESE: Catalog = Catalog::new("ja", "日本語", include_str!(concat!(env!("OUT_DIR"), "/locales/ja.json")));
static KOREAN: Catalog = Catalog::new("ko", "한국어", include_str!(concat!(env!("OUT_DIR"), "/locales/ko.json")));
static MALAY: Catalog = Catalog::new("ms", "Bahasa Melayu", include_str!(concat!(env!("OUT_DIR"), "/locales/ms.json")));
static DUTCH: Catalog = Catalog::new("nl", "Nederlands", include_str!(concat!(env!("OUT_DIR"), "/locales/nl.json")));
static POLISH: Catalog = Catalog::new("pl", "Polski", include_str!(concat!(env!("OUT_DIR"), "/locales/pl.json")));
static PORTUGUESE_BR: Catalog = Catalog::new("pt-BR", "Português (Brasil)", include_str!(concat!(env!("OUT_DIR"), "/locales/pt-BR.json")));
static ROMANIAN: Catalog = Catalog::new("ro", "Română", include_str!(concat!(env!("OUT_DIR"), "/locales/ro.json")));
static RUSSIAN: Catalog = Catalog::new("ru", "Русский", include_str!(concat!(env!("OUT_DIR"), "/locales/ru.json")));
static SWEDISH: Catalog = Catalog::new("sv", "Svenska", include_str!(concat!(env!("OUT_DIR"), "/locales/sv.json")));
static TURKISH: Catalog = Catalog::new("tr", "Türkçe", include_str!(concat!(env!("OUT_DIR"), "/locales/tr.json")));
static UKRAINIAN: Catalog = Catalog::new("uk", "Українська", include_str!(concat!(env!("OUT_DIR"), "/locales/uk.json")));
static VIETNAMESE: Catalog = Catalog::new("vi", "Tiếng Việt", include_str!(concat!(env!("OUT_DIR"), "/locales/vi.json")));
static CHINESE: Catalog = Catalog::new("zh", "中文", include_str!(concat!(env!("OUT_DIR"), "/locales/zh.json")));
static CHINESE_TRADITIONAL: Catalog = Catalog::new("zh-TW", "繁體中文", include_str!(concat!(env!("OUT_DIR"), "/locales/zh-TW.json")));
/// English first (the fallback), then by language id, one per line so that adding a language
/// doesn't edit a line another addition edits.
#[rustfmt::skip]
@@ -267,8 +263,22 @@ pub fn tr_name(text: &str) -> String {
mod tests {
use super::*;
/// Strings a pull request adds under `assets/locales/pending/` work before they are folded
/// into the catalogs (the build merges them in).
#[test]
fn pending_messages_are_in_the_catalogs() {
let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../assets/locales/pending");
for entry in std::fs::read_dir(dir).unwrap().filter_map(Result::ok).filter(|e| e.path().extension().is_some_and(|x| x == "json")) {
let pending: BTreeMap<String, Messages> = serde_json::from_str(&std::fs::read_to_string(entry.path()).unwrap()).unwrap();
for (locale, messages) in pending {
for (id, text) in messages {
assert_eq!(Locale::for_language(&locale).text(&id), text, "{locale}: {id}");
}
}
}
}
#[test]
fn alias_resolution_regression() {
static LEGACY: Catalog = Catalog::new("test-legacy", "Test", r#"{"Pitch":"legacy pitch"}"#);
static ALIAS: Catalog = Catalog::new("test-alias", "Test", r#"{"coupling.pitch":"mechanical pitch"}"#);
+1 -1
View File
@@ -16,7 +16,7 @@ use solidcraft_engine::Session;
/// Every English text `label()` can be asked to translate.
fn english_texts() -> BTreeSet<String> {
const EN: &str = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/en.json"));
const EN: &str = include_str!(concat!(env!("OUT_DIR"), "/locales/en.json"));
let messages: serde_json::Map<String, serde_json::Value> = serde_json::from_str(EN).expect("en.json is valid JSON");
messages.values().filter_map(|v| v.as_str().map(str::to_string)).collect()
}
+1 -1
View File
@@ -13,7 +13,7 @@ use crate::{Services, SolveApp};
/// English text → the catalog ids that carry it.
fn english_ids() -> BTreeMap<String, Vec<String>> {
const EN: &str = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../assets/locales/en.json"));
const EN: &str = include_str!(concat!(env!("OUT_DIR"), "/locales/en.json"));
let messages: BTreeMap<String, String> = serde_json::from_str(EN).unwrap();
let mut ids: BTreeMap<String, Vec<String>> = BTreeMap::new();
for (id, text) in messages {
+1 -1
View File
@@ -372,7 +372,7 @@ mod tests {
"moisture_unit",
];
for catalog in crate::i18n::LOCALES.iter().filter(|c| c.id != "en-GB") {
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../assets/locales").join(format!("{}.json", catalog.id));
let path = std::path::Path::new(env!("OUT_DIR")).join("locales").join(format!("{}.json", catalog.id));
let messages: Value = serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap();
for key in keys.into_iter().chain(KINDS) {
let key = format!("ui.material_review.{key}");
+2 -1
View File
@@ -667,7 +667,8 @@ mod tests {
#[test]
fn promoted_long_labels_have_compact_ribbon_names() {
let specs = solidcraft_engine::command_specs();
let english: std::collections::BTreeMap<String, String> = serde_json::from_str(include_str!("../../../assets/locales/en.json")).unwrap();
let english: std::collections::BTreeMap<String, String> =
serde_json::from_str(include_str!(concat!(env!("OUT_DIR"), "/locales/en.json"))).unwrap();
for section in SECTIONS {
let (commands, count) = layout(section.tab, section.panel, &specs).unwrap();
for command in commands.iter().take(count) {
+9 -1
View File
@@ -58,6 +58,9 @@ fn walk(dir: &Path, root: &Path, out: &mut Vec<String>) -> Result<(), String> {
fn needs_attribution(path: &str) -> bool {
// Include code masters, native work, catalogs, metadata and licence texts in the collection.
if path.starts_with("assets/locales/pending/") {
return false;
}
if path.starts_with("assets/") {
return !path.ends_with(".md");
}
@@ -71,7 +74,12 @@ fn missing<'a>(files: &'a [String], attribution: &str) -> Vec<&'a str> {
}
pub fn run(root: &Path) -> Result<(), String> {
let attribution = std::fs::read_to_string(root.join("ATTRIBUTION.md")).map_err(|e| format!("ATTRIBUTION.md: {e}"))?;
let mut attribution = std::fs::read_to_string(root.join("ATTRIBUTION.md")).map_err(|e| format!("ATTRIBUTION.md: {e}"))?;
// Rows still pending (attribution/pending/*.md) count: they are folded on main.
for (_, row) in crate::pending::attribution_rows(root)? {
attribution.push('\n');
attribution.push_str(&row);
}
let mut files = Vec::new();
walk(root, root, &mut files)?;
files.sort();
+8 -1
View File
@@ -122,8 +122,15 @@ fn locales(root: &Path) -> Result<Vec<String>, String> {
fn read(root: &Path, locale: &str) -> Result<Map<String, Value>, String> {
let path = root.join(DIR).join(format!("{locale}.json"));
let text = std::fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display()))?;
// Messages still in assets/locales/pending/ count: they are folded on main.
let pending = crate::pending::locale_messages(root, locale)?;
match serde_json::from_str(&text) {
Ok(Value::Object(map)) => Ok(map),
Ok(Value::Object(mut map)) => {
for (id, text) in pending {
map.entry(id).or_insert(Value::String(text));
}
Ok(map)
}
Ok(_) => Err(format!("{}: a catalog must be a JSON object", path.display())),
Err(e) => Err(format!("{}: {e}", path.display())),
}
+3 -1
View File
@@ -37,7 +37,9 @@ pub fn run(root: &Path, fix: bool) -> Result<(), String> {
stale.join(", ")
));
}
println!("lists: message catalogs are sorted");
crate::pending::check_locales(root)?;
crate::pending::check_attribution(root)?;
println!("lists: message catalogs are sorted; pending locale and attribution files fold cleanly");
Ok(())
}
+5
View File
@@ -12,6 +12,7 @@ mod icons;
mod layers;
mod licences;
mod lists;
mod pending;
mod render_assets;
mod test;
mod test_nextest;
@@ -55,6 +56,9 @@ commands:
write THIRD-PARTY-LICENSES.txt (the licences of every shipped crate, embedded
in the programs); --check only reports when it is out of date: pull requests
leave it to the Generated files workflow on main
fold-pending fold assets/locales/pending/*.json into the catalogs and attribution/pending/*.md
into ATTRIBUTION.md, and delete them (the Generated files workflow runs this on
main; pull requests add the pending files instead of editing the shared ones)
lists [--fix] check that files many changes add to are sorted one item per line (the
message catalogs in assets/locales); --fix sorts them
test [CARGO ARGS] [-- TEST ARGS]
@@ -92,6 +96,7 @@ fn main() -> ExitCode {
}
Some("book") => book::run(&root(), !rest.contains(&"--no-build")),
Some("licences") => licences::run(&root(), rest.contains(&"--check")),
Some("fold-pending") => pending::run(&root()),
Some("lists") => lists::run(&root(), rest.contains(&"--fix")),
Some("test") => {
let (cargo_args, test_args) = rest.iter().position(|a| *a == "--").map_or((&rest[..], &[][..]), |i| (&rest[..i], &rest[i + 1..]));
+292
View File
@@ -0,0 +1,292 @@
//! Pending files: one file per change for what many pull requests add to, folded on `main`.
//!
//! - `assets/locales/pending/<slug>.json`: `{"<locale>": {"<id>": "<text>"}}`, folded into the
//! catalogs (`assets/locales/<locale>.json`, sorted as `cargo xtask lists` expects).
//! - `attribution/pending/<slug>.md`: rows of the ATTRIBUTION.md files table, appended to it.
//!
//! `cargo xtask fold-pending` folds and deletes them (the Generated files workflow runs it on
//! `main`); `check_*` is what `cargo xtask lists`, `i18n` and `assets` run on every pull request.
//! Both read `README.md` in each folder as documentation, never as a pending file.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use serde_json::{Map, Value};
const LOCALES: &str = "assets/locales";
const LOCALES_PENDING: &str = "assets/locales/pending";
const ATTRIBUTION_PENDING: &str = "attribution/pending";
/// One pending locale file: its path and, per locale, the messages it adds.
type LocalePending = (PathBuf, BTreeMap<String, BTreeMap<String, String>>);
fn files_with(dir: &Path, ext: &str) -> Result<Vec<PathBuf>, String> {
let Ok(entries) = std::fs::read_dir(dir) else { return Ok(Vec::new()) };
let mut files: Vec<PathBuf> = entries
.filter_map(Result::ok)
.map(|e| e.path())
.filter(|p| p.extension().is_some_and(|x| x == ext) && p.file_name().is_some_and(|n| n != "README.md"))
.collect();
files.sort();
Ok(files)
}
fn show(root: &Path, path: &Path) -> String {
path.strip_prefix(root).unwrap_or(path).display().to_string()
}
fn read_locale_pending(root: &Path) -> Result<Vec<LocalePending>, String> {
let mut out = Vec::new();
for path in files_with(&root.join(LOCALES_PENDING), "json")? {
let name = show(root, &path);
let text = std::fs::read_to_string(&path).map_err(|e| format!("{name}: {e}"))?;
let map: BTreeMap<String, BTreeMap<String, String>> =
serde_json::from_str(&text).map_err(|e| format!("{name}: expected {{\"<locale>\": {{\"<id>\": \"<text>\"}}}}: {e}"))?;
out.push((path, map));
}
Ok(out)
}
fn read_catalog(root: &Path, locale: &str) -> Result<Map<String, Value>, String> {
let path = root.join(LOCALES).join(format!("{locale}.json"));
let text = std::fs::read_to_string(&path).map_err(|e| format!("{}: {e}", show(root, &path)))?;
match serde_json::from_str(&text) {
Ok(Value::Object(map)) => Ok(map),
Ok(_) => Err(format!("{}: a catalog must be a JSON object", show(root, &path))),
Err(e) => Err(format!("{}: {e}", show(root, &path))),
}
}
/// The messages the pending files add to one locale, for the tools that read the catalogs.
pub fn locale_messages(root: &Path, locale: &str) -> Result<BTreeMap<String, String>, String> {
let mut out = BTreeMap::new();
for (_, map) in read_locale_pending(root)? {
if let Some(messages) = map.get(locale) {
out.extend(messages.clone());
}
}
Ok(out)
}
/// Fails on a pending file the fold could not apply: a locale without a catalog, an id the catalog
/// already has, or an id two pending files both add. Returns nothing else.
pub fn check_locales(root: &Path) -> Result<(), String> {
plan_locales(root).map(|_| ())
}
type Plan = (BTreeMap<String, Map<String, Value>>, Vec<PathBuf>);
/// Locale → catalog with the pending messages folded in, plus the pending files that fed them.
fn plan_locales(root: &Path) -> Result<Plan, String> {
let pending = read_locale_pending(root)?;
let mut catalogs: BTreeMap<String, Map<String, Value>> = BTreeMap::new();
let mut added_by: BTreeMap<(String, String), String> = BTreeMap::new();
let mut problems = Vec::new();
for (path, map) in &pending {
let name = show(root, path);
for (locale, messages) in map {
if !root.join(LOCALES).join(format!("{locale}.json")).is_file() {
problems.push(format!("{name}: `{locale}` has no catalog assets/locales/{locale}.json"));
continue;
}
if !catalogs.contains_key(locale) {
catalogs.insert(locale.clone(), read_catalog(root, locale)?);
}
for (id, text) in messages {
if let Some(other) = added_by.get(&(locale.clone(), id.clone())) {
problems.push(format!("{name}: `{id}` ({locale}) is also added by {other}"));
} else if catalogs[locale].contains_key(id) {
problems.push(format!("{name}: `{id}` is already in {locale}.json (rename the new id, or edit the catalog)"));
} else {
added_by.insert((locale.clone(), id.clone()), name.clone());
if let Some(catalog) = catalogs.get_mut(locale) {
catalog.insert(id.clone(), Value::String(text.clone()));
}
}
}
}
}
if problems.is_empty() { Ok((catalogs, pending.into_iter().map(|(p, _)| p).collect())) } else { Err(problems.join("\n")) }
}
/// Fold every pending locale file into the catalogs and delete it. Returns how many messages.
pub fn fold_locales(root: &Path) -> Result<usize, String> {
let (catalogs, files) = plan_locales(root)?;
if files.is_empty() {
return Ok(0);
}
let mut count = 0;
for (locale, catalog) in &catalogs {
let before = read_catalog(root, locale)?.len();
count += catalog.len() - before;
let text = crate::lists::canonical(&serde_json::to_string(catalog).map_err(|e| e.to_string())?)?;
let path = root.join(LOCALES).join(format!("{locale}.json"));
std::fs::write(&path, text).map_err(|e| format!("{}: {e}", show(root, &path)))?;
}
for file in files {
std::fs::remove_file(&file).map_err(|e| format!("{}: {e}", show(root, &file)))?;
}
Ok(count)
}
/// The first cell of a table row (the file path), when the line is a data row of the files table.
fn row_file(line: &str) -> Option<&str> {
let line = line.trim();
if !line.starts_with('|') {
return None;
}
let cell = line.split('|').nth(1)?.trim();
(!cell.is_empty() && cell != "File" && !cell.starts_with("---") && !cell.starts_with(":-")).then_some(cell)
}
/// The rows of every pending attribution file: (file name, row line). Anything else in a pending
/// file besides the table header and blank lines is an error, so a stray note is not lost.
pub fn attribution_rows(root: &Path) -> Result<Vec<(String, String)>, String> {
let mut rows = Vec::new();
for path in files_with(&root.join(ATTRIBUTION_PENDING), "md")? {
let name = show(root, &path);
let text = std::fs::read_to_string(&path).map_err(|e| format!("{name}: {e}"))?;
for line in text.lines().map(str::trim_end).filter(|l| !l.trim().is_empty()) {
if row_file(line).is_some() {
rows.push((name.clone(), line.to_string()));
} else if !line.trim_start().starts_with('|') {
return Err(format!("{name}: only ATTRIBUTION.md table rows (`| File | Author | Source | Licence |`) belong here, not `{line}`"));
}
}
}
Ok(rows)
}
/// ATTRIBUTION.md with the pending rows appended to its files table; an error when a row's file
/// already has one (in the table or in another pending file).
fn merged_attribution(root: &Path) -> Result<Option<String>, String> {
let rows = attribution_rows(root)?;
if rows.is_empty() {
return Ok(None);
}
let path = root.join("ATTRIBUTION.md");
let text = std::fs::read_to_string(&path).map_err(|e| format!("ATTRIBUTION.md: {e}"))?;
let mut lines: Vec<&str> = text.lines().collect();
let start = lines.iter().position(|l| l.trim() == "## Files").ok_or("ATTRIBUTION.md has no `## Files` section")?;
let end = lines[start..].iter().position(|l| l.starts_with("## ") && l.trim() != "## Files").map_or(lines.len(), |i| start + i);
let last_row = (start..end).rev().find(|&i| lines[i].trim_start().starts_with('|')).ok_or("ATTRIBUTION.md: no files table under `## Files`")?;
let mut seen: BTreeMap<String, String> =
lines[start..end].iter().filter_map(|l| row_file(l)).map(|f| (f.to_string(), "ATTRIBUTION.md".into())).collect();
let mut problems = Vec::new();
for (name, row) in &rows {
let Some(file) = row_file(row) else { continue };
if let Some(other) = seen.get(file) {
problems.push(format!("{name}: `{file}` already has a row in {other}"));
} else {
seen.insert(file.to_string(), name.clone());
}
}
if !problems.is_empty() {
return Err(problems.join("\n"));
}
let new: Vec<&str> = rows.iter().map(|(_, r)| r.as_str()).collect();
lines.splice(last_row + 1..last_row + 1, new);
Ok(Some(lines.join("\n") + "\n"))
}
pub fn check_attribution(root: &Path) -> Result<(), String> {
merged_attribution(root).map(|_| ())
}
/// Append the pending rows to ATTRIBUTION.md and delete the pending files. Returns the row count.
pub fn fold_attribution(root: &Path) -> Result<usize, String> {
let Some(text) = merged_attribution(root)? else { return Ok(0) };
let count = attribution_rows(root)?.len();
std::fs::write(root.join("ATTRIBUTION.md"), text).map_err(|e| format!("ATTRIBUTION.md: {e}"))?;
for file in files_with(&root.join(ATTRIBUTION_PENDING), "md")? {
std::fs::remove_file(&file).map_err(|e| format!("{}: {e}", show(root, &file)))?;
}
Ok(count)
}
/// `cargo xtask fold-pending`.
pub fn run(root: &Path) -> Result<(), String> {
let messages = fold_locales(root)?;
let rows = fold_attribution(root)?;
println!("fold-pending: {messages} message(s) into assets/locales, {rows} row(s) into ATTRIBUTION.md");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn tree(catalogs: &[(&str, &str)]) -> PathBuf {
use std::sync::atomic::{AtomicUsize, Ordering};
static N: AtomicUsize = AtomicUsize::new(0);
let root = std::env::temp_dir().join(format!("xtask-pending-{}-{}", std::process::id(), N.fetch_add(1, Ordering::Relaxed)));
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(root.join(LOCALES_PENDING)).unwrap();
std::fs::create_dir_all(root.join(ATTRIBUTION_PENDING)).unwrap();
for (locale, text) in catalogs {
std::fs::write(root.join(LOCALES).join(format!("{locale}.json")), text).unwrap();
}
std::fs::write(
root.join("ATTRIBUTION.md"),
"# Attribution\n\n## Files\n\n| File | Author | Source | Licence |\n|---|---|---|---|\n| a.png | me | own | MIT |\n\n## Other\n\ntext\n",
)
.unwrap();
root
}
fn put(root: &Path, rel: &str, text: &str) {
std::fs::write(root.join(rel), text).unwrap();
}
#[test]
fn locale_files_fold_sorted_and_are_removed() {
let root = tree(&[("en", "{\n \"b\": \"B\"\n}\n"), ("ja", "{\n \"b\": \"ビ\"\n}\n")]);
put(&root, "assets/locales/pending/one.json", r#"{"en": {"z": "Z", "a": "A"}, "ja": {"a": "ア"}}"#);
put(&root, "assets/locales/pending/two.json", r#"{"en": {"m": "M"}}"#);
put(&root, "assets/locales/pending/README.md", "docs, never folded");
assert_eq!(locale_messages(&root, "en").unwrap().len(), 3);
assert_eq!(fold_locales(&root).unwrap(), 4);
let en = std::fs::read_to_string(root.join("assets/locales/en.json")).unwrap();
assert_eq!(en, "{\n \"a\": \"A\",\n \"b\": \"B\",\n \"m\": \"M\",\n \"z\": \"Z\"\n}\n");
assert_eq!(en, crate::lists::canonical(&en).unwrap());
assert!(std::fs::read_to_string(root.join("assets/locales/ja.json")).unwrap().contains("\"a\": \"ア\""));
assert!(!root.join("assets/locales/pending/one.json").exists() && !root.join("assets/locales/pending/two.json").exists());
assert!(root.join("assets/locales/pending/README.md").exists());
assert!(locale_messages(&root, "en").unwrap().is_empty());
}
#[test]
fn a_key_the_catalog_has_or_two_files_add_fails_and_nothing_is_changed() {
let root = tree(&[("en", "{\n \"b\": \"B\"\n}\n")]);
put(&root, "assets/locales/pending/one.json", r#"{"en": {"b": "again"}}"#);
let err = fold_locales(&root).unwrap_err();
assert!(err.contains("`b` is already in en.json"), "{err}");
assert!(root.join("assets/locales/pending/one.json").exists());
assert_eq!(std::fs::read_to_string(root.join("assets/locales/en.json")).unwrap(), "{\n \"b\": \"B\"\n}\n");
put(&root, "assets/locales/pending/one.json", r#"{"en": {"c": "C"}}"#);
put(&root, "assets/locales/pending/two.json", r#"{"en": {"c": "C2"}}"#);
assert!(check_locales(&root).unwrap_err().contains("is also added by"));
put(&root, "assets/locales/pending/two.json", r#"{"xx": {"d": "D"}}"#);
assert!(check_locales(&root).unwrap_err().contains("no catalog"));
put(&root, "assets/locales/pending/two.json", "[]");
assert!(check_locales(&root).is_err());
}
#[test]
fn attribution_rows_fold_into_the_files_table_and_a_repeat_fails() {
let root = tree(&[]);
put(&root, "attribution/pending/x.md", "| File | Author | Source | Licence |\n|---|---|---|---|\n| b.png | me | own | MIT |\n");
put(&root, "attribution/pending/y.md", "| c.png | me | own | MIT |\n");
assert_eq!(attribution_rows(&root).unwrap().len(), 2);
assert_eq!(fold_attribution(&root).unwrap(), 2);
let text = std::fs::read_to_string(root.join("ATTRIBUTION.md")).unwrap();
assert!(text.contains("| a.png | me | own | MIT |\n| b.png | me | own | MIT |\n| c.png | me | own | MIT |\n\n## Other"), "{text}");
assert!(!root.join("attribution/pending/x.md").exists());
put(&root, "attribution/pending/z.md", "| a.png | you | own | MIT |\n");
assert!(check_attribution(&root).unwrap_err().contains("already has a row"));
put(&root, "attribution/pending/z.md", "a stray note\n");
assert!(check_attribution(&root).is_err());
}
}