Model cache: each sheet body is written once, not with every state (#6152)
* Test: a saved model keeps each sheet body once A sheet-metal design wrote its sheet bodies again with every later feature's state in the saved model: each feature that leaves the sheet alone (a box beside it) adds more than the whole sheet to the records. A design of many features after its sheet metal passed the cache's 32 MiB manifest limit that way and kept only some of its states (#3220). The test also opens the saved file and wants the sheets taken from it as evaluated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsiqyXWBFGTCX9J51bhysc * Model cache: each sheet body is written once, not with every state Problem: a saved model writes every feature's state, and each state wrote its sheet bodies in full; most features leave the sheets as they were, so a sheet-metal design of many features repeated them over and over (a 280-feature part: 30.5 MB of manifest, 29.6 MB of it 69 distinct sheet records written 243 times). Past the manifest's 32 MiB limit the save thinned the cache (#3220) and opening evaluated the left-out features again, although the distinct data was small. Solution: the records keep a list of sheet bodies, each written once; a state names its sheets by their place in it, as it already does for bodies and sketches. A thinned model keeps the whole list and counts it once against its budget; the reader counts sheets with the bodies and sketches a state holds; the cache format is now /4 (an older cache is a miss). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsiqyXWBFGTCX9J51bhysc --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,7 +22,7 @@ pub struct SavedModel {
|
||||
|
||||
/// Records refer to these at most.
|
||||
const MAX_ITEMS: usize = 1_000_000;
|
||||
/// Bodies and sketches all the states hold together, at most (a state is a list of clones).
|
||||
/// Bodies, sketches and sheet bodies all the states hold together, at most (a state is a list of clones).
|
||||
const MAX_HELD: usize = 50_000_000;
|
||||
|
||||
/// A feature's boolean retries as saved. The kernel leaves `rejected` out when it is empty, and
|
||||
@@ -118,6 +118,10 @@ struct Writer {
|
||||
names_index: HashMap<usize, usize>,
|
||||
sketches: Vec<Json>,
|
||||
sketch_index: HashMap<usize, usize>,
|
||||
/// Sheet bodies by their record: most features leave the sheets as they were, so the states
|
||||
/// share them (writing them with every state made a sheet-metal design's model too large to keep).
|
||||
sheets: Vec<Json>,
|
||||
sheet_index: HashMap<String, usize>,
|
||||
states: Vec<Json>,
|
||||
state_index: HashMap<usize, usize>,
|
||||
}
|
||||
@@ -184,6 +188,17 @@ impl Writer {
|
||||
Some(self.sketches.len() - 1)
|
||||
}
|
||||
|
||||
fn sheet(&mut self, sheet: &crate::sheet::SheetBody) -> Option<usize> {
|
||||
let record = serde_json::to_value(sheet).ok()?;
|
||||
let key = record.to_string();
|
||||
if let Some(i) = self.sheet_index.get(&key) {
|
||||
return Some(*i);
|
||||
}
|
||||
self.sheets.push(record);
|
||||
self.sheet_index.insert(key, self.sheets.len() - 1);
|
||||
Some(self.sheets.len() - 1)
|
||||
}
|
||||
|
||||
fn state(&mut self, st: &Arc<ModelState>) -> Option<usize> {
|
||||
let id = Arc::as_ptr(st) as usize;
|
||||
if let Some(i) = self.state_index.get(&id) {
|
||||
@@ -191,10 +206,11 @@ impl Writer {
|
||||
}
|
||||
let bodies = st.bodies.iter().map(|b| self.model_body(b)).collect::<Option<Vec<_>>>()?;
|
||||
let sketches = st.sketches.iter().map(|s| self.sketch(s)).collect::<Option<Vec<_>>>()?;
|
||||
let sheets = st.sheets.iter().map(|s| self.sheet(s)).collect::<Option<Vec<_>>>()?;
|
||||
let r = json!({
|
||||
"bodies": runs(&bodies), "sketches": runs(&sketches), "body_counter": st.body_counter,
|
||||
"reserved_body_names": st.reserved_body_names.iter().collect::<Vec<_>>(),
|
||||
"sheets": serde_json::to_value(&st.sheets).ok()?, "construct": serde_json::to_value(&st.construct).ok()?,
|
||||
"sheets": sheets, "construct": serde_json::to_value(&st.construct).ok()?,
|
||||
"threads": serde_json::to_value(&st.threads).ok()?,
|
||||
"sketch_keys": st.sketch_keys.iter().map(|(f, k)| json!([f, hex(*k)])).collect::<Vec<_>>(),
|
||||
"blend_edges": serde_json::to_value(&st.blend_edges).ok()?, "fitted": serde_json::to_value(&st.fitted).ok()?,
|
||||
@@ -230,6 +246,7 @@ impl Model {
|
||||
}
|
||||
let records = json!({
|
||||
"features": features, "states": w.states, "model_bodies": w.model_bodies, "names": w.names, "sketches": w.sketches,
|
||||
"sheets": w.sheets,
|
||||
});
|
||||
Ok(SavedModel { records, bodies: w.bodies })
|
||||
})
|
||||
@@ -306,6 +323,10 @@ impl SavedModel {
|
||||
}
|
||||
model_bodies.push(b);
|
||||
}
|
||||
let mut sheets = Vec::new();
|
||||
for s in list("sheets")? {
|
||||
sheets.push(serde_json::from_value::<crate::sheet::SheetBody>(s.clone()).map_err(|_| bad("saved model: sheet"))?);
|
||||
}
|
||||
let mut states = Vec::new();
|
||||
let mut held = 0usize;
|
||||
for s in list("states")? {
|
||||
@@ -339,7 +360,13 @@ impl SavedModel {
|
||||
.and_then(Json::as_array)
|
||||
.map(|a| a.iter().filter_map(|n| n.as_str().map(str::to_string)).collect())
|
||||
.unwrap_or_default(),
|
||||
sheets: serde_json::from_value(field("sheets")?).map_err(invalid)?,
|
||||
sheets: s
|
||||
.get("sheets")
|
||||
.and_then(Json::as_array)
|
||||
.ok_or_else(|| bad("saved model: state"))?
|
||||
.iter()
|
||||
.map(|i| index(Some(i), sheets.len()).and_then(|i| sheets.get(i).cloned().ok_or_else(|| bad("saved model: sheet"))))
|
||||
.collect::<Result<Vec<_>>>()?,
|
||||
construct: serde_json::from_value(field("construct")?).map_err(invalid)?,
|
||||
threads: serde_json::from_value(field("threads")?).map_err(invalid)?,
|
||||
sketch_keys,
|
||||
@@ -347,7 +374,7 @@ impl SavedModel {
|
||||
fitted: serde_json::from_value(field("fitted")?).map_err(invalid)?,
|
||||
tapered: serde_json::from_value(field("tapered")?).map_err(invalid)?,
|
||||
};
|
||||
held = held.saturating_add(st.bodies.len()).saturating_add(st.sketches.len());
|
||||
held = held.saturating_add(st.bodies.len()).saturating_add(st.sketches.len()).saturating_add(st.sheets.len());
|
||||
if held > MAX_HELD {
|
||||
return Err(bad("saved model: states"));
|
||||
}
|
||||
@@ -424,7 +451,9 @@ impl SavedModel {
|
||||
let mut kept_bodies = vec![false; model_bodies.len()];
|
||||
let (mut names_used, mut sketches_used) = (vec![false; names.len()], vec![false; sketches.len()]);
|
||||
let mut stored = std::collections::HashSet::new();
|
||||
let mut used = 0usize;
|
||||
// The sheet bodies, each written once and named by place in the states: all kept.
|
||||
let sheets = list("sheets").cloned().unwrap_or_default();
|
||||
let mut used = bytes(sheets.as_slice()).iter().fold(0usize, |t, b| t.saturating_add(*b));
|
||||
for s in std::iter::once(last).chain(order) {
|
||||
let (mut adds, mut new_names, mut new_sketches) = (HashMap::new(), std::collections::HashSet::new(), std::collections::HashSet::new());
|
||||
let mut records = *state_bytes.get(s)?;
|
||||
@@ -516,7 +545,7 @@ impl SavedModel {
|
||||
}
|
||||
let records = json!({
|
||||
"features": out_features, "states": out_states, "model_bodies": out_bodies, "names": kept_of(names.as_slice(), &names_used),
|
||||
"sketches": kept_of(sketches.as_slice(), &sketches_used),
|
||||
"sketches": kept_of(sketches.as_slice(), &sketches_used), "sheets": sheets,
|
||||
});
|
||||
let kept: Vec<usize> = (0..self.bodies.len()).filter(|i| kernel_used.get(*i) == Some(&true)).collect();
|
||||
let bodies = kept.iter().filter_map(|i| self.bodies.get(*i).cloned()).collect();
|
||||
|
||||
@@ -923,6 +923,49 @@ mod tests {
|
||||
std::fs::remove_file(path).unwrap();
|
||||
}
|
||||
|
||||
/// A sheet-metal design keeps each sheet body once in its saved model, not again with every
|
||||
/// later feature's state: a design of many features after its sheet metal wrote the sheets
|
||||
/// over and over, and past the cache's size limit kept only some of its states.
|
||||
#[test]
|
||||
fn a_saved_model_keeps_each_sheet_body_once() {
|
||||
let design = |boxes: usize| {
|
||||
let mut s = Session::new(Document::new("Sheet"));
|
||||
let run = |s: &mut Session, id: &str, p: Value| s.execute(id, &p).unwrap();
|
||||
run(
|
||||
&mut s,
|
||||
"sheet.manage_rules",
|
||||
json!({"name":"AL2","thickness":"2 mm","k_factor":0.5,"bend_radius":"1 mm","gap":"0.5 mm","active":true}),
|
||||
);
|
||||
run(&mut s, "sketch.create", json!({"plane":"XY","name":"Base"}));
|
||||
run(&mut s, "sketch.rectangle.two_point", json!({"p0":[0,0],"p1":[200,150]}));
|
||||
run(&mut s, "sketch.finish", json!({}));
|
||||
run(&mut s, "sheet.flange", json!({"sketch":"Base"}));
|
||||
run(&mut s, "sheet.flange", json!({"edges":[[100,0,2],[200,75,2],[100,150,2],[0,75,2]],"height":"20 mm","angle":"90 deg"}));
|
||||
// A panel's holes: each one is a cut-out the sheet keeps.
|
||||
for i in 0..16 {
|
||||
run(&mut s, "solid.hole", json!({"position":[25 + 10 * i,40,2],"diameter":5}));
|
||||
}
|
||||
for i in 0..boxes {
|
||||
run(&mut s, "solid.box", json!({"length":5,"width":5,"height":5,"corner":[300 + 10 * i,0,0],"operation":"new"}));
|
||||
}
|
||||
s
|
||||
};
|
||||
let size = |s: &Session| serde_json::to_vec(&saved_geometry(s, &s.doc).unwrap().records).unwrap().len();
|
||||
let (few, many) = (design(2), design(12));
|
||||
let sheet = serde_json::to_vec(&many.model.state().sheets).unwrap().len();
|
||||
let per_feature = (size(&many) - size(&few)) / 10;
|
||||
assert!(per_feature * 2 < sheet, "each later feature adds {per_feature} bytes, the sheet is {sheet}");
|
||||
// Taken from the file as saved: the sheet is the one evaluated.
|
||||
let flat = many.model.state().sheets.clone();
|
||||
let path = std::env::temp_dir().join(format!("solidcraft-sheet-cache-{}.solidcraft", std::process::id()));
|
||||
std::fs::write(&path, many.design_container(solidcraft_io::DesignCompression::Fast, false, true).unwrap()).unwrap();
|
||||
let mut reopened = Session::default();
|
||||
reopened.execute("doc.open", &json!({"path":path})).unwrap();
|
||||
assert!(reopened.model.results.iter().zip(&many.model.results).all(|(a, b)| a.ms == b.ms), "taken from the file, not evaluated");
|
||||
assert_eq!(serde_json::to_value(&reopened.model.state().sheets).unwrap(), serde_json::to_value(&flat).unwrap());
|
||||
std::fs::remove_file(path).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore = "manual native cache save benchmark; generate issue303-step.json first"]
|
||||
fn benchmark_native_cache_save() {
|
||||
|
||||
@@ -954,7 +954,7 @@ mod tests {
|
||||
solidcraft_doc::SavedModel {
|
||||
records: json!({
|
||||
"features": [{"id": 1, "fingerprint": "0000000000000000", "ms": 0.0, "skipped": false, "state": 0}],
|
||||
"states": [state], "model_bodies": records, "names": [{"faces": [], "info": []}], "sketches": [],
|
||||
"states": [state], "model_bodies": records, "names": [{"faces": [], "info": []}], "sketches": [], "sheets": [],
|
||||
}),
|
||||
bodies,
|
||||
}
|
||||
@@ -1060,7 +1060,7 @@ mod tests {
|
||||
.map(|(i, ms)| json!({"id": i + 1, "fingerprint": "0000000000000000", "ms": ms, "skipped": false, "state": i}))
|
||||
.collect();
|
||||
solidcraft_doc::SavedModel {
|
||||
records: json!({"features": features, "states": states, "model_bodies": records, "names": [{"faces": [], "info": []}], "sketches": []}),
|
||||
records: json!({"features": features, "states": states, "model_bodies": records, "names": [{"faces": [], "info": []}], "sketches": [], "sheets": []}),
|
||||
bodies,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
use super::*;
|
||||
use solidcraft_doc::{ModelState, SavedModel};
|
||||
|
||||
const FORMAT: &str = "solidcraft-model-cache/3";
|
||||
const FORMAT: &str = "solidcraft-model-cache/4";
|
||||
const PRODUCER: &str = "solidcraft-truck-adapter/3";
|
||||
const MANIFEST: &str = "cache/model.json";
|
||||
const PAYLOAD: &str = "cache/bodies.bin";
|
||||
|
||||
@@ -147,7 +147,7 @@ SHA-256 of their encoding), and a placed copy of a repeated part (an instance fr
|
||||
a STEP assembly) records the part and its matrix, so it shows that part's display
|
||||
again instead of being meshed on its own.
|
||||
|
||||
`cache/model.json` records `solidcraft-model-cache/3`, the producer revision
|
||||
`cache/model.json` records `solidcraft-model-cache/4`, the producer revision
|
||||
`solidcraft-truck-adapter/3`, the application version, the build (source commit,
|
||||
CPU architecture and OS), the SHA-256 of the exact `document.json` (whose blob
|
||||
references pin every STEP, mesh and image payload) and of `cache/bodies.bin`, the
|
||||
@@ -168,7 +168,9 @@ A model whose cache would push the container past them keeps fewer states
|
||||
other states, the slowest feature's first (the evaluations opening would otherwise
|
||||
repeat for longest), each while what it adds still fits: the bodies no state kept
|
||||
so far holds (a body is stored once however many states hold it) and the records
|
||||
of the state, those bodies, their names and sketches. When the rest of the
|
||||
of the state, those bodies, their names and sketches. Sheet bodies, like sketches, are
|
||||
written once and named by place in the states (most features leave the sheets as they were); a
|
||||
thinned cache keeps all of them. When the rest of the
|
||||
manifest (the bodies' table, the slices) then leaves too little room, the choice
|
||||
is made again within that much less, and the final state alone is always tried.
|
||||
A feature whose state is left out records none. The cache is left out (and the
|
||||
|
||||
Reference in New Issue
Block a user