fix: review findings in the editor, the cell memory and use() memo

- use(): two notebooks run under one graph id replace each other's memo entry (the registry holds one run's nodes
  per graph id; the older entry handed out a product whose nodes were gone).
- Editor: a post that is not a readable preview is answered 400, not 409; the server's model is one value replaced
  atomically; the signing of a publish is handed out once, so a stray second post cannot be filed under it.
- Cell memory: set_metadata/auto_tag_faces change a value in place, so an entry holding a value they changed is
  dropped (it was handed out changed); the marimo hook checks the signature it re-implements and says so when it
  cannot; the length reads under the lock.
- Handled failures print a [sca] line (preview warm-up, lineage record, cache entry delete, preview frame, signing,
  damaged kept model) instead of passing silently.
- Tests: a damaged preview, bytes that are no preview, the single model value, the signing handed out once, a
  moved and turned solid keeps its location and face orientation through .wrapped, saved values in the digests of a
  real notebook, the marimo hook, a value changed in place.
- Docs: docs/environment.md lists every SCA_* switch; CHANGELOG states the one-time rebuild of notebooks with saved
  parameters, the copy semantics of .wrapped, the memory limit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
NiJingzhe
2026-10-29 02:05:58 +08:00
parent a20d3c54e0
commit ac1b50b990
14 changed files with 435 additions and 67 deletions
+21 -5
View File
@@ -28,37 +28,53 @@ user-facing notes live in [`docs/updates/`](docs/updates/) (English and
its cache belongs to.
### Changed
- A handled failure that used to leave no trace now prints a `[sca] ...` line on stderr: the preview warm-up, an
unreadable lineage record, a cache entry that could not be deleted, a preview frame that could not be built, a
notebook that could not be signed, and a kept model that is damaged and is built again.
- `shape.wrapped` is an OCP shape when OCP is installed (the `verify` extra), of the class of its kind (`TopoDS_Solid`,
`TopoDS_Face`, ...), as it is in CadQuery: the OCCT code an agent writes for verifier scripts runs as written
(`BRepGProp.VolumeProperties_s(solid.wrapped, props)`). It is a copy made through BinTools; without OCP it stays the
SDK's own OCCT shape. `SCA_WRAPPED=native|ocp` chooses. The SDK's own Python layers take `shape._native_wrapped`, which
(`BRepGProp.VolumeProperties_s(solid.wrapped, props)`). It is a copy made through BinTools, with its location and
orientation: every access makes a new one (`s.wrapped is s.wrapped` is false), so a change made to it (a
`BRepBuilderAPI_Transform`, `BRepTools.Clean`) does not reach the SDK's shape, which is unlike CadQuery's live handle
and unlike 2.2.6, where `.wrapped` was the shape itself. To use a result, convert it back with
`simplecadapi._glue.bridge.ocp_to_native`. Without OCP it stays the SDK's own OCCT shape. `SCA_WRAPPED=native|ocp`
chooses, and is read on each access. The SDK's own Python layers take `shape._native_wrapped`, which
is always the native one, and the functions that take a raw shape (the `inspect.brep` queries, the STEP export, the
geometry helpers of `simplecadapi.kernel`) accept either.
- Editor kernel: the cells keep what they made (`simplecadapi.runtime.cache._CellMemory`, 12 results; `SCA_NO_CELL_MEMORY=1`
turns it off). marimo unpickles an entry while it answers the cache lookup, which is about a second for each of the
six-axis arm's assemblies; the kernel now asks its memory right after the cell's key is hashed and before the disk,
and hands out the same objects when the entry is one this process holds. Only plain data and the SDK's value types are
kept (they are functional: a cell cannot change what an earlier one made). `SCA_EDITOR_TRACE=1` with
kept. They are functional except for `set_metadata` and `auto_tag_faces`, which change the value they are called on:
those two are watched, and an entry holding a value that one of them changed is dropped, so the next restore reads
the disk copy as before (a change made through a part of an assembly, `assembly.get_component(...)`, is not seen).
The limit is twelve entries, not a number of bytes, and costs about 300 MB on the six-axis arm; only the editor's
kernel has a memory (`sca run` and the tests do not). With a marimo whose `cache_attempt_from_hash` has another
signature than the one this was written for, a line on stderr says so and the memory is off. `SCA_EDITOR_TRACE=1` with
`SCA_EDITOR_TRACE_FILE=<path>` writes what the cache did with each cell and how long each part took.
- The saved parameter values in a cell's digest follow what the cell reads. A value the notebook's parameter cell
defines used to be part of the digest of every cell below that cell, so changing one joint angle changed the digest
of the cell that builds the parts and every cell after it. It is now part of the digest of the cell that defines it,
of the cells that read it, and of the cells that read from those. On the six-axis arm a joint angle that has not been
seen before takes 2.3-2.7 s to show instead of 5.7-6.5 s, and one that has been seen takes 0.6 s instead of 5.5 s.
The digests of cells of a notebook that has saved parameter values are not those of the previous version, so the
first run after the upgrade builds those notebooks again once.
- Editor: the preview is incremental. A model's parts are identified by the content of their geometry, so the kernel
posts only the parts the server does not hold, the server sends each page only the parts that page lacks (a page that
joins late, or asks with `preview.resync`, is given the whole model), and the page builds geometry only for new
parts and reuses the rest. On the six-axis arm a joint angle now moves 115 KB instead of 6.26 MB: the kernel's post
takes 17 ms instead of 285-610 ms, the page handles the frame in 2-5 ms with no long task, and from the last cell
to the model on screen it takes 0.36-0.41 s instead of 1.3-3.2 s. `simplecadapi.scene.preview.encode_preview` takes
`omit=` for the parts to leave out; the editor keeps parts in `simplecadapi.editor.preview_parts.PartStore`.
`omit=` for the parts to leave out; a post that is not a readable preview is answered `400` (`409` is only for parts
the server does not hold), and the server replaces the model it holds as one value; the editor keeps parts in `simplecadapi.editor.preview_parts.PartStore`.
- Editor: the signature that files a model on disk is taken beside the build, not before it, and the server replies to
the kernel's post before filing the model and writing the snapshot. `SCA_EDITOR_TRACE=1` prints how long each step
of a settle takes.
- `use` keeps the product of the last run of each child notebook in the process and hands it out again while the
child's static digest (source, header, local modules, input files, the children it uses) and the overrides are
unchanged (`simplecadapi.runtime.usememo`; `SCA_NO_USE_MEMO=1` turns it off). One entry per (notebook, id): the run's
nodes are registered by graph id, so the entry is the last run's. The editor's kernel is a long-lived process, so a
nodes are registered by graph id, so the entry is the last run's, and two notebooks that run under one graph id
replace each other's entry. The editor's kernel is a long-lived process, so a
new `OPEN_STROKE` on the six-axis arm rebuilds in 4.2 s instead of 6.4 s. Children with a computed `use` path and
overrides that are not plain data are not kept. Prefetch no longer starts workers for what is kept.
- Editor: the model is pushed over the WebSocket with the `preview` event, in one binary frame
+24
View File
@@ -0,0 +1,24 @@
# Environment switches
The SDK reads these variables. The first group are kill switches for the editor's speed-ups: each turns one of them
off, to tell a speed-up from a bug (a result that differs with the switch set is a bug; please report it).
| Variable | Effect |
|---|---|
| `SCA_NO_CELL_MEMORY=1` | The editor kernel does not keep what its cells made; every restore reads the disk cache. |
| `SCA_NO_USE_MEMO=1` | `use()` runs the child notebook each time instead of handing out the product of its last run. |
| `SCA_NO_PREVIEW_WARM=1` | Parts are not tessellated in the background while the next cell computes. |
| `SCA_NO_KERNEL_CACHE=1` | The editor kernel's cells are not cached. |
| `SCA_NO_DIAGNOSTIC_RENDER=1` | A failed operation does not render its diagnostic picture. |
| Variable | Effect |
|---|---|
| `SCA_WRAPPED=native\|ocp` | What `shape.wrapped` returns. Default: an OCP shape when OCP is installed (the `verify` extra), else the SDK's own OCCT shape. Read on each access. |
| `SCA_EDITOR_TRACE=1` | The editor prints how long each step of a settle takes, and what the cache did with each cell. |
| `SCA_EDITOR_TRACE_FILE=<path>` | With `SCA_EDITOR_TRACE`, write the trace there instead of stderr. |
| `SCA_EDITOR_CALL_TIMEOUT`, `SCA_EDITOR_OPEN_WAIT` | Seconds the editor waits for a kernel call / for the page to open. |
| `SCA_JOBS` | Worker processes for building child notebooks ahead of `use()`. |
| `SCA_DIAGNOSTIC_RENDER_BUDGET` | Seconds a failure's diagnostic render may take. |
| `SCA_RUNTIME_DIR`, `SCA_ADDON_DIR`, `SCA_ADDON_HOME`, `SCA_SKILLS_DIR`, `SCA_TARGET` | Where the runtime state, add-ons, skills and the install target live. |
A handled failure that must still be seen prints a `[sca] ...` line on stderr; nothing is swallowed silently.
+20
View File
@@ -0,0 +1,20 @@
"""A line on stderr for a failure that is handled and must still be seen (never a silent ``except``)."""
from __future__ import annotations
import sys
import threading
_said: set[str] = set()
_lock = threading.Lock()
def say(message: str, *, once: str | None = None) -> None:
"""Print ``[sca] message``; with *once*, only the first time that key is used in this process."""
if once is not None:
with _lock:
if once in _said:
return
_said.add(once)
print(f"[sca] {message}", file=sys.stderr, flush=True)
+26 -17
View File
@@ -46,26 +46,35 @@ class ParsedPreview:
def parse_preview(data: bytes) -> ParsedPreview:
"""Split an encoded preview (full or with parts omitted) into nodes and per-part blocks."""
"""Split an encoded preview (full or with parts omitted) into nodes and per-part blocks.
Raises :class:`PreviewError` for anything that is not a readable preview: bytes without the magic, a header that is
cut off or not JSON, a schema this reader does not know, a part that lacks a field.
"""
if not data.startswith(MAGIC):
raise PreviewError("not a preview")
(length,) = struct.unpack_from("<I", data, len(MAGIC))
start = len(MAGIC) + 8
header = json.loads(data[start : start + length])
if header.get("schema") != PREVIEW_SCHEMA:
raise PreviewError(f"unsupported preview schema: {header.get('schema')!r}")
base = start + length
parts: dict[str, Stored] = {}
for part in header["parts"]:
first = part["positions"]["offset"]
last = part["vertex_positions"]
end = last["offset"] + last["count"] * _ITEM
meta = dict(part)
for name in _ARRAYS:
meta[name] = {"offset": part[name]["offset"] - first, "count": part[name]["count"]}
parts[part["key"]] = Stored(meta, bytes(data[base + first : base + end]))
return ParsedPreview(list(header["nodes"]), parts)
try:
(length,) = struct.unpack_from("<I", data, len(MAGIC))
start = len(MAGIC) + 8
header = json.loads(data[start : start + length])
if header.get("schema") != PREVIEW_SCHEMA:
raise PreviewError(f"unsupported preview schema: {header.get('schema')!r}")
base = start + length
parts: dict[str, Stored] = {}
for part in header["parts"]:
first = part["positions"]["offset"]
last = part["vertex_positions"]
end = last["offset"] + last["count"] * _ITEM
meta = dict(part)
for name in _ARRAYS:
meta[name] = {"offset": part[name]["offset"] - first, "count": part[name]["count"]}
parts[part["key"]] = Stored(meta, bytes(data[base + first : base + end]))
return ParsedPreview(list(header["nodes"]), parts)
except PreviewError:
raise
except (struct.error, ValueError, KeyError, TypeError, AttributeError) as error:
raise PreviewError(f"damaged preview: {type(error).__name__}: {error}") from error
class PartStore:
+57 -20
View File
@@ -18,15 +18,15 @@ import ast
import asyncio
import hashlib
import json
import re
import os
import re
import sys
import threading
import time
import webbrowser
from collections import OrderedDict
from pathlib import Path
from typing import Any
from typing import Any, NamedTuple
import uvicorn
from starlette.applications import Starlette
@@ -53,7 +53,7 @@ from ..runtime.workspace import features_in_cell, notebook_modules, scan_modules
from .kernel import MarimoKernel
from .selection import MARKER, selection_cell_code, variable_names
from .templates import create_notebook
from ..scene.preview import PreviewError
from ..scene.preview import MAGIC, PreviewError
from .preview_parts import PartStore, parse_preview
from .ws import WsHub, preview_frame
@@ -84,6 +84,14 @@ def pick_notebook(workspace: Path, prefer: Path | None = None) -> Path | None:
return (workspace / chosen.path).resolve()
class _Model(NamedTuple):
"""The model the pages show, as one value: what it is made of, the part keys, and the bytes of all of it."""
nodes: list[dict] | None
keys: frozenset[str]
data: bytes | None
class _Signing:
"""The signature being taken for one publish, and when it is done."""
@@ -124,11 +132,10 @@ class WorkspaceState:
self.lock = threading.Lock()
self.latest: dict[str, dict] = {}
self.latest_package_bytes: bytes | None = None # a full package, made on demand (export)
self.latest_preview_bytes: bytes | None = None
# The parts of the latest preview by key, and what it is made of: a page is sent only the parts it lacks.
# The latest preview (replaced as a whole, so a reader on another thread never sees half of one) and the
# parts of it by key: a page is sent only the parts it lacks.
self._model = _Model(None, frozenset(), None)
self.part_store = PartStore()
self.scene_nodes: list[dict] | None = None
self.latest_keys: frozenset[str] = frozenset()
self.preview_revision = 0
self._package_event = threading.Event()
self.selection: dict = {}
@@ -151,6 +158,7 @@ class WorkspaceState:
self._packages: OrderedDict[str, tuple[str, bytes]] = OrderedDict()
self._publish_signature: str | None = None
self._signing: _Signing | None = None
self._signing_lock = threading.Lock()
self._stopped = False
self._selection_code: str | None = None
self._selection_busy = False
@@ -227,8 +235,7 @@ class WorkspaceState:
self.notebook = target
# Let the webuis drop the old model now; commands have no kernel until the new one is up.
self.latest.clear()
self.latest_preview_bytes = None
self.scene_nodes = None
self._model = _Model(None, frozenset(), None)
self._drop_kernel()
return True
@@ -281,8 +288,7 @@ class WorkspaceState:
self._last_publish_error = ""
if not restored:
self.latest.clear()
self.latest_preview_bytes = None
self.scene_nodes = None
self._model = _Model(None, frozenset(), None)
self.cells = []
self.specs = {}
self._watched = self._watched_files()[0]
@@ -344,7 +350,7 @@ class WorkspaceState:
try:
parsed = parse_preview(data)
except PreviewError:
return True
return not data.startswith(MAGIC) # (a damaged preview is no model; bytes that are not one are kept as they are)
return parsed.needed <= parsed.parts.keys()
def remember_model(self, notebook: Path, signature: str, data: bytes) -> None:
@@ -395,12 +401,14 @@ class WorkspaceState:
return False
if not self._is_whole(data):
# (a file an older build kept, or a damaged one): nothing to show, and nothing to start the kernel on
print(f"[sca edit] the kept model of {notebook.name} is not usable; building it again", file=sys.stderr)
return False
self.notebook = notebook
self._publish_cells(_static_cells(notebook))
try:
self.set_preview(data)
except (KeyError, ValueError):
except (KeyError, PreviewError) as error:
print(f"[sca edit] the kept model of {notebook.name} is not usable ({error}); building it again", file=sys.stderr)
return False
self.preview_revision += 1
self.emit(
@@ -957,18 +965,43 @@ class WorkspaceState:
else:
self.hub.broadcast(envelope)
@property
def latest_preview_bytes(self) -> bytes | None:
return self._model.data
@latest_preview_bytes.setter
def latest_preview_bytes(self, data: bytes | None) -> None:
self._model = _Model(None, frozenset(), data) # bytes that are not split into parts
@property
def scene_nodes(self) -> list[dict] | None:
return self._model.nodes
@property
def latest_keys(self) -> frozenset[str]:
return self._model.keys
def take_signing(self) -> "_Signing | None":
"""The signing of the publish that is arriving, handed out once: a second post under it has none to file under."""
with self._signing_lock:
signing, self._signing = self._signing, None
return signing
def set_preview(self, data: bytes) -> bool:
"""Take *data* (a full preview, or one with parts the store already holds) as the latest model.
Returns whether it is the model already held. Bytes that are not a preview are kept as they are. Raises
``KeyError`` (with the missing part keys) when a part is neither in *data* nor in the store.
``KeyError`` (with the missing part keys) when a part is neither in *data* nor in the store, and
:class:`PreviewError` when *data* announces itself as a preview and is not a readable one.
"""
try:
parsed = parse_preview(data)
except PreviewError:
# Not a preview this server can split (a tool's own bytes): kept and sent whole.
if data.startswith(MAGIC):
raise # a preview that is damaged, or of a schema this server does not know: not bytes to keep
# Not a preview at all (a tool's own bytes): kept and sent whole.
unchanged = self.latest_preview_bytes == data
self.scene_nodes, self.latest_keys, self.latest_preview_bytes = None, frozenset(), bytes(data)
self.latest_preview_bytes = bytes(data)
return unchanged
missing = self.part_store.missing(parsed.needed - parsed.parts.keys())
if missing:
@@ -977,7 +1010,7 @@ class WorkspaceState:
self.part_store.touch(parsed.needed)
full = self.part_store.assemble(parsed.nodes)
unchanged = self.latest_preview_bytes == full
self.scene_nodes, self.latest_keys, self.latest_preview_bytes = parsed.nodes, frozenset(parsed.needed), full
self._model = _Model(parsed.nodes, frozenset(parsed.needed), full)
return unchanged
def full_preview_frame(self, envelope: dict) -> bytes | None:
@@ -1028,7 +1061,8 @@ class WorkspaceState:
notebook = self.notebook
self._publish_signature = None
signing = _Signing()
self._signing = signing
with self._signing_lock:
self._signing = signing
if notebook is None:
signing.done.set()
return
@@ -1037,8 +1071,9 @@ class WorkspaceState:
began = time.perf_counter()
try:
signing.value = self._signature(notebook)
except (OSError, ValueError):
except (OSError, ValueError) as error:
signing.value = None
print(f"[sca edit] could not sign the notebook ({error}); the model will not be filed", file=sys.stderr)
if self._signing is signing:
self._publish_signature = signing.value
signing.done.set()
@@ -1889,6 +1924,8 @@ def create_app(state: WorkspaceState) -> Starlette:
arrived = time.perf_counter()
try:
unchanged = state.set_preview(bytes(body))
except PreviewError as error:
return JSONResponse({"ok": False, "error": str(error)}, status_code=400)
except KeyError as missing:
# Parts the kernel left out are not here (the server was given another model since): it sends them all.
return JSONResponse({"ok": False, "missing": missing.args[0]}, status_code=409)
@@ -1909,7 +1946,7 @@ def create_app(state: WorkspaceState) -> Starlette:
hub.broadcast({"type": "status", "error": ""}) # the model is fine again: clear a stale "no product yet"
threading.Thread(
target=state.after_publish,
args=(arrived, len(body), state._signing, state.latest_preview_bytes),
args=(arrived, len(body), state.take_signing(), state.latest_preview_bytes),
name="after-publish",
daemon=True,
).start()
+17 -5
View File
@@ -13,6 +13,8 @@ from typing import Any, Callable, Collection
from starlette.websockets import WebSocket
from .._glue.notice import say
class WsHub:
def __init__(self) -> None:
@@ -50,10 +52,16 @@ class WsHub:
clients = tuple(self._clients.items())
for client, known in clients:
try:
await client.send_bytes(make_frame(frozenset(known)))
frame = make_frame(frozenset(known))
except Exception as error: # a frame that cannot be made is this server's fault, not the page's
say(f"could not build a preview frame ({type(error).__name__}: {error}); the page keeps its model")
continue
try:
await client.send_bytes(frame)
except Exception: # the page went away
await self._drop(client)
else:
self._clients[client] = set(keys)
except Exception:
self._clients.pop(client, None)
def broadcast(self, message: dict[str, Any]) -> None:
"""Push one JSON message to every connected client (thread-safe)."""
@@ -82,8 +90,12 @@ class WsHub:
await client.send_bytes(frame)
else:
await client.send_text(frame)
except Exception:
self._clients.pop(client, None)
except Exception: # the page went away
await self._drop(client)
async def _drop(self, client: Any) -> None:
async with self._lock:
self._clients.pop(client, None)
def preview_frame(envelope: dict[str, Any], data: bytes) -> bytes:
+75 -7
View File
@@ -22,6 +22,7 @@ use, so going back to a previous value is still instant.
from __future__ import annotations
import hashlib
import inspect
import json
import os
import shutil
@@ -29,10 +30,11 @@ import threading
import time
import zlib
from collections import OrderedDict
from collections.abc import Callable, Collection, Mapping, Sequence
from collections.abc import Callable, Collection, Iterator, Mapping, Sequence
from pathlib import Path
from typing import Any
from .._glue.notice import say
from ._marimo import (
CachedLifecycle,
CellId_t,
@@ -212,7 +214,10 @@ class CellDigests:
_MEMORY_ENTRIES = 12
"""How many cell results the kernel keeps (the newest; an assembly of a few dozen parts is some tens of megabytes)."""
"""How many cell results the kernel keeps (the newest; counted in entries, not bytes: an assembly of a few dozen parts is
some tens of megabytes, which is where the editor's extra resident memory comes from). Only an editor kernel has a
memory (``CellCache(memory=True)``); ``sca run`` and the tests do not, and the SDK's own code cannot change under it:
the kernel is a process that imported it once."""
def _shareable(value: Any) -> bool:
@@ -230,6 +235,53 @@ def _shareable(value: Any) -> bool:
return kind.__module__ == "simplecadapi.core" or kind.__module__.startswith("simplecadapi.product.")
_MUTATORS = ("set_metadata", "auto_tag_faces")
_changed: set[int] = set()
"""Ids of values something changed in place (a later cell calling ``set_metadata`` on a value it was given)."""
def _watch_changes() -> None:
"""Note which value a ``set_metadata``/``auto_tag_faces`` call changes, for every SDK type that has them.
The SDK's values are functional except for these: they change the value they are called on, and a memory that hands
the same object to a later run would hand out the change (the disk copy is a fresh object each time). The ids are
kept in :data:`_changed`; a memory entry holding one of them is not handed out again (a wrong id only costs a read).
"""
import functools
from .. import _cpp
native = getattr(_cpp, "_native", None)
for kind in vars(native).values() if native is not None else ():
if not isinstance(kind, type):
continue
for name in _MUTATORS:
original = kind.__dict__.get(name)
if original is None or getattr(original, "_sca_watched", False):
continue
def watched(self, *args, __original=original, **kwargs):
if len(_changed) > 100_000:
_changed.clear()
_changed.add(id(self))
return __original(self, *args, **kwargs)
functools.update_wrapper(watched, original)
watched._sca_watched = True # type: ignore[attr-defined]
setattr(kind, name, watched)
_watch_changes()
def _held(value: Any) -> "Iterator[Any]":
yield value
if isinstance(value, (tuple, frozenset)):
for item in value:
yield from _held(item)
class _CellMemory:
"""The values of cells restored or run in this process, by cache entry.
@@ -253,6 +305,9 @@ class _CellMemory:
entry = self._entries.get(key)
if entry is None or entry[0] is not loader: # (an id is reused once its loader is gone)
return None
if any(id(held) in _changed for value in entry[1].values() for held in _held(value)):
del self._entries[key] # something changed a value in place since it was kept
return None
self._entries.move_to_end(key)
return entry[1], entry[2]
@@ -261,6 +316,7 @@ class _CellMemory:
if stateful or not all(_shareable(value) for value in values.values()):
return
key = self._key(loader, key)
_changed.difference_update(id(held) for value in values.values() for held in _held(value))
with self._lock:
self._entries[key] = (loader, values, meta)
self._entries.move_to_end(key)
@@ -272,7 +328,8 @@ class _CellMemory:
self._entries.clear()
def __len__(self) -> int:
return len(self._entries)
with self._lock:
return len(self._entries)
def _attempt_with_memory(
@@ -338,7 +395,17 @@ def _attempt_with_memory(
_marimo_attempt = _marimo_cached.cache_attempt_from_hash
_marimo_cached.cache_attempt_from_hash = _attempt_with_memory # a loader without a memory is not affected
_MEMORY_NEEDS = {"module", "graph", "cell_id", "scope", "context", "pin_modules", "hash_type", "scoped_refs", "loader", "as_fn"}
if _MEMORY_NEEDS <= set(inspect.signature(_marimo_attempt).parameters):
_marimo_cached.cache_attempt_from_hash = _attempt_with_memory # a loader without a memory is not affected
else:
# The cell memory re-implements the first half of this marimo function; with another signature it cannot.
say(
"this marimo's cache_attempt_from_hash has another signature than the cell memory was written for: "
"the editor kernel will not keep cell results in memory (every restore reads the disk)",
once="cell-memory-marimo",
)
class CellCache(CachedLifecycle):
@@ -447,8 +514,8 @@ class CellCache(CachedLifecycle):
try:
manifest.unlink(missing_ok=True)
shutil.rmtree(blobs, ignore_errors=True)
except OSError:
pass
except OSError as error:
say(f"could not delete the stale cache entry {manifest} ({error}); it will be read again")
def _load_lineage(self, cell: CellImpl, digest: str) -> bool:
from . import lineage
@@ -456,7 +523,8 @@ class CellCache(CachedLifecycle):
try:
# The entries of a run live in <state>/<id>/cache: the nodes of its cells belong to the graph of that id.
return lineage.load(self._root / digest, str(cell.cell_id), self._root.parent.name)
except Exception: # noqa: BLE001 - an unreadable record is a miss, never a failure
except Exception as error: # noqa: BLE001 - an unreadable record is a miss, never a failure
say(f"unreadable lineage record of cell {cell.cell_id} ({type(error).__name__}: {error}); running it again")
return False
+6 -1
View File
@@ -201,7 +201,12 @@ def use(path: str | Path, /, *, id: str | None = None, **overrides: Any) -> Prod
if child.product is None:
raise RuntimeError(f"{child.config.path}: the child run produced no product")
record_dependencies({child.config.path, *child.dependencies})
usememo.keep(target, id, overrides, usememo.Kept(child.product, child.config.path, frozenset(child.dependencies)))
usememo.keep(
target,
id,
overrides,
usememo.Kept(child.product, child.config.path, frozenset(child.dependencies), child.config.id),
)
return child.product
+10 -2
View File
@@ -35,9 +35,11 @@ class Kept:
product: Any
path: Path
dependencies: frozenset[Path]
graph_id: str
_entries: "OrderedDict[tuple[str, str | None], tuple[tuple[str, str], Kept]]" = OrderedDict()
_owners: dict[str, tuple[str, str | None]] = {} # graph id -> the entry whose run registered its nodes
_lock = threading.Lock()
@@ -77,15 +79,21 @@ def keep(target: Path, id: str | None, overrides: Mapping[str, Any], kept: Kept)
signature = _signature(target, overrides)
key = (str(target.resolve()), id)
with _lock:
# The registry now holds this run's nodes: whatever entry owned the graph id before no longer matches.
previous = _owners.pop(kept.graph_id, None)
if previous is not None:
_entries.pop(previous, None)
if signature is None:
_entries.pop(key, None) # the registry now holds this run's nodes: an older entry no longer matches
_entries.pop(key, None)
return
_entries[key] = (signature, kept)
_entries.move_to_end(key)
_owners[kept.graph_id] = key
while len(_entries) > _MAX_ENTRIES:
_entries.popitem(last=False)
_owners.pop(_entries.popitem(last=False)[1][1].graph_id, None)
def clear() -> None:
with _lock:
_entries.clear()
_owners.clear()
+5 -3
View File
@@ -16,6 +16,7 @@ import threading
from typing import Any, Iterable, Iterator
from simplecadapi._cpp import Assembly, Part, Solid
from simplecadapi._glue.notice import say
from .preview import preview_solid
@@ -59,7 +60,8 @@ def submit(values: Iterable[Any]) -> int:
except queue.Full:
break
queued += 1
except Exception: # noqa: BLE001 - an optimisation: whatever a cell defined, it must not fail the cell
except Exception as error: # noqa: BLE001 - an optimisation: whatever a cell defined, it must not fail the cell
say(f"preview warm-up skipped a value ({type(error).__name__}: {error})", once="prewarm-submit")
return queued
if queued:
_ensure_worker()
@@ -82,5 +84,5 @@ def _run() -> None:
return # the thread ends when idle; the next submit starts another
try:
preview_solid(solid)
except Exception: # noqa: BLE001 - the preview that follows reports a part that cannot be drawn
pass
except Exception as error: # noqa: BLE001 - the preview that follows reports a part that cannot be drawn
say(f"preview warm-up could not draw a part ({type(error).__name__}: {error})", once="prewarm-run")
+79 -4
View File
@@ -10,7 +10,11 @@ pytest.importorskip("marimo")
import simplecadapi as scad
from simplecadapi.runtime import cache as cache_module
from simplecadapi.runtime.cache import _CellMemory, _attempt_with_memory, _shareable
from simplecadapi.runtime._marimo import InternalApp, load_notebook
from simplecadapi.runtime.config import read_notebook_config
from simplecadapi.runtime.runner import _cell_prefix
from test_notebook_runtime import NotebookTestCase, cell, notebook_text
from simplecadapi.runtime.cache import CellDigests, _CellMemory, _attempt_with_memory, _shareable
class Loader:
@@ -146,9 +150,6 @@ class MemoryTest(unittest.TestCase):
self.assertIsNotNone(self.memory.recall(self.loader, Key("c")))
if __name__ == "__main__":
unittest.main()
class SavedValuesFollowWhatACellReadsTest(unittest.TestCase):
"""A saved value is in the digest of the cells it reaches, not of every cell below the cell that defines it."""
@@ -192,3 +193,77 @@ class SavedValuesFollowWhatACellReadsTest(unittest.TestCase):
def test_nothing_saved_means_nothing_to_carry(self) -> None:
self.assertTrue(all(not found for found in self.names(set()).values()))
class SavedValuesInTheDigestsOfARealNotebookTest(NotebookTestCase):
"""What the editor applies to a variable changes the digest of the cells that get it, through any chain of cells."""
def digests(self, saved):
path = self.write(
"arm.py",
notebook_text(
cell("A = 1.0\nB = 2.0", returns="A, B"),
cell("half = A / 2", reads="A", returns="half"),
cell("quarter = half / 2", reads="half", returns="quarter"),
cell("other = B + 1", reads="B", returns="other"),
),
)
config = read_notebook_config(path)
internal = InternalApp(load_notebook(path, cell_prefix=_cell_prefix(config)))
ordered = [internal.graph.cells[i] for i in internal.cell_manager.cell_ids() if i in internal.graph.cells]
setup = internal.graph.cells.get(internal.cell_manager.setup_cell_id)
found = CellDigests(config, ordered, setup, saved=saved, kernel=True)
return {cell_.code.split(" =")[0].split("\n")[0]: found.digest(cell_.cell_id) for cell_ in ordered}
def test_a_value_reaches_a_cell_that_only_reads_it_through_another(self) -> None:
before, after = self.digests({"A": 1.0}), self.digests({"A": 5.0})
self.assertNotEqual(before["half"], after["half"])
self.assertNotEqual(before["quarter"], after["quarter"]) # reads `half`, which reads `A`
self.assertEqual(before["other"], after["other"]) # reads only `B`
def test_the_same_saved_values_give_the_same_digests(self) -> None:
self.assertEqual(self.digests({"A": 5.0}), self.digests({"A": 5.0}))
class MarimoHookTest(unittest.TestCase):
def test_marimo_asks_the_function_the_memory_replaced(self) -> None:
"""The cell memory re-implements the first half of marimo's ``cache_attempt_from_hash``: a marimo whose
signature changed (the upgrade this would break on) has to fail here, not quietly lose the memory."""
import inspect
from simplecadapi.runtime import cache
from simplecadapi.runtime._marimo import marimo_cached
self.assertTrue(cache._MEMORY_NEEDS <= set(inspect.signature(cache._marimo_attempt).parameters))
self.assertIs(marimo_cached.cache_attempt_from_hash, cache._attempt_with_memory)
class AValueAnotherCellChangedIsNotHandedOutTest(unittest.TestCase):
"""``set_metadata`` and ``auto_tag_faces`` change a value in place: what a cell did to a value it was given must
not be seen by a later run that restores the entry (the disk copy is a fresh object each time)."""
def setUp(self) -> None:
self.memory = _CellMemory(limit=4)
self.loader = object()
def test_a_value_changed_in_place_after_it_was_kept_is_a_miss(self) -> None:
import simplecadapi as scad
box = scad.make_box_rsolid(2, 2, 2)
self.memory.remember(self.loader, Key("e"), {"box": box}, {})
self.assertIsNotNone(self.memory.recall(self.loader, Key("e")))
box.set_metadata("note", "changed by a later cell")
self.assertIsNone(self.memory.recall(self.loader, Key("e")))
self.assertEqual(len(self.memory), 0) # and the entry is gone, not just skipped
def test_a_value_kept_again_after_a_change_is_the_new_state(self) -> None:
import simplecadapi as scad
box = scad.make_box_rsolid(2, 2, 2)
box.set_metadata("note", "before")
self.memory.remember(self.loader, Key("e"), {"box": box}, {})
self.assertIsNotNone(self.memory.recall(self.loader, Key("e")))
if __name__ == "__main__":
unittest.main()
+44 -3
View File
@@ -151,6 +151,46 @@ class ServerSendsPagesWhatTheyLackTest(unittest.TestCase):
self.assertTrue(state.set_preview(P.encode_preview(scene, omit=set(scene.parts)))) # same model, parts left out
self.assertEqual(state.latest_preview_bytes, full)
def test_a_damaged_preview_is_refused_as_such_not_as_missing_parts(self) -> None:
from starlette.testclient import TestClient
from simplecadapi.editor.server import create_app
state = self.state()
client = TestClient(create_app(state))
scene = scene_of(scad.make_box_rsolid(10, 10, 10))
full = P.encode_preview(scene)
for name, body in {
"cut inside the header": full[: len(P.MAGIC) + 8 + 5],
"header that is not JSON": full[: len(P.MAGIC) + 8] + b"x" * (len(full) - len(P.MAGIC) - 8),
}.items():
with self.subTest(name):
reply = client.post("/api/runtime/preview", content=body)
self.assertEqual(reply.status_code, 400, reply.text)
self.assertNotIn("missing", reply.json())
self.assertIsNone(state.latest_preview_bytes) # nothing was taken
self.assertEqual(client.post("/api/runtime/preview", content=full).status_code, 200)
def test_bytes_that_are_no_preview_at_all_are_still_kept_whole(self) -> None:
state = self.state()
self.assertFalse(state.set_preview(b"PKG"))
self.assertEqual(state.latest_preview_bytes, b"PKG")
def test_the_model_is_replaced_as_one_value(self) -> None:
state = self.state()
scene = scene_of(scad.make_box_rsolid(10, 10, 10))
state.set_preview(P.encode_preview(scene))
nodes, keys, data = state.scene_nodes, state.latest_keys, state.latest_preview_bytes
self.assertEqual(state._model, (nodes, keys, data))
self.assertEqual(keys, frozenset(scene.parts))
def test_the_signing_of_a_publish_is_handed_out_once(self) -> None:
state = self.state()
state._sign_in_background()
first = state.take_signing()
self.assertIsNotNone(first)
self.assertIsNone(state.take_signing()) # a second post under the same settle has no signature to be filed under
def test_a_connected_page_is_sent_only_new_parts_and_a_late_page_everything(self) -> None:
from starlette.testclient import TestClient
@@ -181,9 +221,6 @@ class ServerSendsPagesWhatTheyLackTest(unittest.TestCase):
self.assertEqual(set(parse_preview(whole).parts), set(second.parts))
if __name__ == "__main__":
unittest.main()
class HubSendsEachPageItsOwnFrameTest(unittest.TestCase):
def test_each_page_is_given_a_frame_made_for_the_parts_it_holds(self) -> None:
@@ -226,3 +263,7 @@ class HubSendsEachPageItsOwnFrameTest(unittest.TestCase):
time.sleep(0.01)
self.assertEqual(old.sent[1], b"b,c") # and it now holds the scene it was sent
self.assertEqual(new.sent[1], b"b,c")
if __name__ == "__main__":
unittest.main()
+8
View File
@@ -84,6 +84,14 @@ class UseMemoTest(unittest.TestCase):
self.assertEqual(recorded[0], recorded[1])
self.assertIn(self.child.resolve(), {path.resolve() for path in recorded[1]})
def test_two_children_under_one_id_do_not_both_stay_kept(self) -> None:
other = Path(create_notebook(self.root, "other", "part"))
first = runner.use(self.child, id="shared")
runner.use(other, id="shared") # registers its nodes under "shared", replacing the first run's
again = runner.use(self.child, id="shared")
self.assertIsNot(again, first) # the first run's nodes are gone from the registry: it has to run again
self.assertEqual([run[0] for run in self.runs], ["child.py", "other.py", "child.py"])
if __name__ == "__main__":
unittest.main()
+43
View File
@@ -113,3 +113,46 @@ def test_the_way_back_from_an_ocp_shape_keeps_the_geometry() -> None:
def test_something_that_is_not_a_shape_is_refused_with_a_reason() -> None:
with pytest.raises(TypeError, match="SDK shape"):
bridge.native_to_ocp(5)
def test_a_moved_and_turned_solid_keeps_its_place_and_the_orientation_of_its_faces() -> None:
"""The way through BRep must keep the location and the orientation flags, not only the volume and the class."""
from OCP.BRepAdaptor import BRepAdaptor_Surface
from OCP.BRepBndLib import BRepBndLib
from OCP.BRepGProp import BRepGProp
from OCP.BRepTools import BRepTools
from OCP.Bnd import Bnd_Box
from OCP.GProp import GProp_GProps
from OCP.TopAbs import TopAbs_FACE, TopAbs_REVERSED
from OCP.TopExp import TopExp_Explorer
from OCP.TopoDS import TopoDS
from OCP.gp import gp_Pnt, gp_Vec
solid = scad.rotate_shape(scad.translate_shape(box(), (10.0, -5.0, 3.0)), 30.0, axis=(1, 1, 0))
shape = solid.wrapped
centre = GProp_GProps()
BRepGProp.VolumeProperties_s(shape, centre)
inside = centre.CentreOfMass()
expected = solid.get_center()
assert (inside.X(), inside.Y(), inside.Z()) == pytest.approx(tuple(expected), abs=1e-6) # the placement is kept
explorer = TopExp_Explorer(shape, TopAbs_FACE)
reversed_faces = faces = 0
while explorer.More():
face = TopoDS.Face_s(explorer.Current())
faces += 1
reversed_faces += face.Orientation() == TopAbs_REVERSED
u0, u1, v0, v1 = BRepTools.UVBounds_s(face)
surface = BRepAdaptor_Surface(face)
point, du, dv = gp_Pnt(), gp_Vec(), gp_Vec()
surface.D1((u0 + u1) / 2, (v0 + v1) / 2, point, du, dv)
normal = du.Crossed(dv)
if face.Orientation() == TopAbs_REVERSED:
normal.Reverse()
outward = gp_Vec(inside, point)
assert normal.Dot(outward) > 0, "a face of a solid must point away from its inside"
explorer.Next()
assert faces == 6 and 0 < reversed_faces < 6 # (a box built by the kernel has both)
bounds = Bnd_Box()
BRepBndLib.Add_s(shape, bounds)
assert bounds.Get()[0] > -20.0 and bounds.Get()[3] < 20.0