fix(windows): unblock builds — binary opens, guarded dir fsync, evidence-based stale locks

Reported by the OmniSim evaluation of the public u-link example
(2026-09-11): pip install + export.py on Windows died three times in
the build layer before any geometry was built.

- Binary-mode opens: every byte-exact os.open site now sets O_BINARY
  via _internal/os_compat.with_binary_flag. autocrlf=true clones read
  builder sources in text mode, stripped CR, and aborted with a false
  file_input_changed (25579-byte u_link.py read back as 25092); the
  same hazard expanded LF on cache writes and would corrupt CAS
  hashes. Sites: build/dependencies.py, artifacts/feature_graph.py,
  scene/archive.py, cache/store.py (object/record writes, lock
  create), artifacts/interface.py (lock create).
- Directory fsync is POSIX-only: os.open(<dir>)+fsync raised
  PermissionError on Windows in cache/store.py and
  artifacts/interface.py; both now no-op off POSIX via one helper.
- Stale-lock recovery by evidence: a lock whose recorded owner pid is
  provably dead is recovered immediately (unreadable owners still wait
  out the age gate, live owners are never broken). Windows liveness
  probing uses OpenProcess/GetExitCodeProcess — os.kill(pid, 0) maps
  to TerminateProcess there and kills the process being probed.
- Console hardening at import: non-UTF strict streams are reconfigured
  to backslashreplace so a cp1252 console cannot raise
  UnicodeEncodeError from a diagnostic print inside an error handler
  and mask the real failure.
- .gitattributes: byte-exact by default, LF pinned for text formats,
  so builder-source digests stop depending on core.autocrlf.

Tests: test_os_compat.py (15) + dead-owner immediate recovery in
test_cache_store.py; full suite 1799 passed, 5 skipped. ty: zero new
diagnostics.
This commit is contained in:
NiJingzhe
2026-09-11 20:55:14 +08:00
parent f94bd006f5
commit e9d5749f7c
12 changed files with 452 additions and 25 deletions
+25
View File
@@ -0,0 +1,25 @@
# Byte-exact by default: no end-of-line conversion for any file unless it
# opts in below. Builder-source digests (part cache keys) hash working-tree
# bytes, so a core.autocrlf=true checkout must not differ from an LF one.
* -text
# Text formats pin LF so every platform checks out identical bytes.
*.py text eol=lf
*.md text eol=lf
*.rst text eol=lf
*.txt text eol=lf
*.toml text eol=lf
*.json text eol=lf
*.yml text eol=lf
*.yaml text eol=lf
*.cfg text eol=lf
*.ini text eol=lf
*.sh text eol=lf
*.ts text eol=lf
*.js text eol=lf
*.mjs text eol=lf
*.css text eol=lf
*.html text eol=lf
*.svg text eol=lf
.gitignore text eol=lf
.gitattributes text eol=lf
+40
View File
@@ -70,6 +70,46 @@ addon authoring guide (`references/domains/addon-development.md`,
routed from the skill) points exporters at it; the normative JSON
Schemas ship inside the wheel at `simplecadapi/contracts/`.
## Windows build and cache correctness
Reported by the OmniSim evaluation of the public u-link example
(2026-09-11): installing the SDK on Windows and running
`examples/u_link_motor_mount/export.py` hit three build-layer blockers
before any geometry was built. All three are fixed, plus two latent
hazards found alongside:
- **Binary-mode opens.** Every byte-exact `os.open` site now sets
`O_BINARY` through one helper (`_internal/os_compat.py`). A Windows
clone made with `core.autocrlf=true` used to read builder sources in
text mode, strip every CR, and abort with a false
`file_input_changed` before the first feature (measured: a
25579-byte `u_link.py` read back as 25092). The same text-mode
hazard silently expanded LF on cache writes, which would corrupt CAS
object hashes. Covered sites: builder source reads
(`build/dependencies.py`, `artifacts/feature_graph.py`), scene
archive member reads (`scene/archive.py`), and every cache
write/lock create (`cache/store.py`,
`artifacts/interface.py`).
- **Directory fsync is POSIX-only.** `os.open(<dir>) + fsync` raised
`PermissionError` on Windows in two places (cache store, latest-part
state), and the first failure stranded a lock file that made every
retry time out. Directory fsync is now a guarded no-op off POSIX.
- **Stale-lock recovery by evidence.** A lock whose recorded owner pid
is provably dead is recovered immediately; unreadable owners still
wait out the age threshold and live owners are never broken. The
Windows liveness probe uses `OpenProcess`/`GetExitCodeProcess` —
`os.kill(pid, 0)` maps to `TerminateProcess` there and would kill
the very process being probed.
- **Console diagnostics cannot mask failures.** Importing the SDK now
reconfigures non-UTF strict streams to `backslashreplace`: a cp1252
console used to raise `UnicodeEncodeError` from a Chinese-language
warning printed inside an error handler, replacing the real failure
with an unrelated "wire is open" fault.
- A root-cause `.gitattributes` ships with the fix — byte-exact by
default, LF pinned for text formats — so every platform checks out
identical bytes and builder-source digests stop depending on
`core.autocrlf`.
## Housekeeping
- Removed the dead `product-package-2.schema.json` contract: the
+34
View File
@@ -54,6 +54,40 @@ kind 实体;缺失的必需名字=点名大声失败,绝不按几何猜)
路由)把导出器作者指向它;规范 JSON Schema 随 wheel 分发于
`simplecadapi/contracts/`。
## Windows 构建与缓存正确性
由 OmniSim 对公开 u-link 示例的实测(2026-09-11)报告:Windows 上
安装 SDK 并运行 `examples/u_link_motor_mount/export.py`,在生成任何
几何之前就撞上三处构建层阻断。三处已全部修复,另修两个顺带发现的
潜伏雷:
- **二进制模式打开。** 所有逐字节 `os.open` 位点统一经
`_internal/os_compat.py` 的 helper 设置 `O_BINARY`。Windows 上以
`core.autocrlf=true` 检出的克隆此前按文本模式读构建器源码、吃掉
全部 CR,在第一个特征之前就以虚假的 `file_input_changed` 终止
(实测:25579 字节的 `u_link.py` 读回 25092)。同一文本模式隐患
还会在缓存写入时悄悄展开 LF、破坏 CAS 对象哈希。覆盖位点:构建
器源码读取(`build/dependencies.py`、
`artifacts/feature_graph.py`)、场景归档成员读取
(`scene/archive.py`)与全部缓存写入/锁创建(`cache/store.py`、
`artifacts/interface.py`)。
- **目录 fsync 仅限 POSIX。** `os.open(<目录>) + fsync` 此前在
Windows 两处(缓存存储、latest-part 状态)抛 `PermissionError`,
且首次失败会遗留锁文件、令此后每次重试超时。目录 fsync 现在在
非 POSIX 平台守卫为 no-op。
- **按证据回收陈旧锁。** 记录在案的 owner pid 可证明已死时立即
回收;不可读的 owner 仍须等满年龄阈值;活 owner 永不被打断。
Windows 存活探针改用 `OpenProcess`/`GetExitCodeProcess` ——
`os.kill(pid, 0)` 在 Windows 上映射为 `TerminateProcess`,会
杀死被探测的进程本身。
- **控制台诊断不再掩盖故障。** 导入 SDK 时把非 UTF 的 strict 流
重配为 `backslashreplace`:cp1252 控制台此前在错误处理器内打印
中文警告时抛 `UnicodeEncodeError`,把真实故障替换成无关的
"wire is open"。
- 随修复附根因级 `.gitattributes`——默认逐字节、文本格式钉死 LF
——各平台检出字节一致,构建器源码摘要不再依赖
`core.autocrlf`。
## 内务
- 移除死合同 `product-package-2.schema.json`:加载器只按
+9
View File
@@ -1,5 +1,14 @@
"""SimpleCAD API: a simplified OCP-native Python CAD modeling API."""
import sys
from ._internal.os_compat import harden_console_streams
# Non-UTF consoles must escape unencodable diagnostic text instead of
# raising UnicodeEncodeError inside error handlers, where it would mask
# the real failure.
harden_console_streams((sys.stdout, sys.stderr))
from . import exporter, inspect, math, ql, std, surface, translator
from .artifacts import (
ArtifactLimits,
+105
View File
@@ -0,0 +1,105 @@
"""OS-portability primitives for byte-exact file IO and process probes.
Windows defaults differ from POSIX in four ways that must not surface as
corrupted cache bytes or masked failures:
- ``os.open`` without ``O_BINARY`` opens in text mode: ``read()`` strips
every CR and ``write()`` expands every LF, so byte counts and hashes
drift on any checkout made with ``core.autocrlf=true``.
- Opening a directory with ``os.open`` and ``fsync`` is POSIX-only;
Windows raises ``PermissionError``.
- ``os.kill(pid, 0)`` maps to ``TerminateProcess`` on Windows and kills
the very process a liveness probe is asking about.
- Legacy consoles (cp1252) raise ``UnicodeEncodeError`` on non-ASCII
diagnostic output, which can replace the real failure inside an error
handler.
"""
from __future__ import annotations
import os
from pathlib import Path
from typing import Iterable
def with_binary_flag(flags: int) -> int:
"""Return read/write flags forced into binary mode on Windows."""
return flags | getattr(os, "O_BINARY", 0)
def fsync_directory(path: Path) -> None:
"""Durability-sync a directory entry; a no-op off POSIX.
Windows offers no portable directory fsync, so rename durability
there rests on the filesystem journal alone.
"""
if os.name != "posix":
return
descriptor = os.open(path, os.O_RDONLY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
def windows_pid_is_alive(pid: int) -> bool:
"""Liveness probe that cannot terminate its target on Windows."""
import ctypes
windll_factory = getattr(ctypes, "WinDLL", None)
if windll_factory is None:
# Only reachable on Windows; unknown platform stays conservative.
return True
kernel32 = windll_factory("kernel32", use_last_error=True)
process_query_limited_information = 0x1000
still_active = 259
handle = kernel32.OpenProcess(
process_query_limited_information, False, pid
)
if not handle:
# ERROR_INVALID_PARAMETER (87) means "no such process"; access
# denied and anything else mean the process exists. A missing
# probe stub stays conservative and reports the process alive.
get_last_error = getattr(ctypes, "get_last_error", lambda: 0)
return get_last_error() != 87
try:
exit_code = ctypes.c_ulong()
if not kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)):
return True
return exit_code.value == still_active
finally:
kernel32.CloseHandle(handle)
def harden_console_streams(streams: Iterable[object]) -> None:
"""Escape instead of crash when a stream cannot encode a diagnostic.
Only non-UTF streams with strict error handling are reconfigured, so
UTF-8 consoles and host-configured streams keep their behavior; the
real failure stays visible instead of being replaced by a
``UnicodeEncodeError`` from a diagnostic print.
"""
for stream in streams:
reconfigure = getattr(stream, "reconfigure", None)
if reconfigure is None:
continue
encoding = (getattr(stream, "encoding", "") or "")
normalized = encoding.replace("-", "").replace("_", "").lower()
if "utf" in normalized:
continue
try:
reconfigure(errors="backslashreplace")
except (ValueError, OSError):
continue
__all__ = [
"fsync_directory",
"harden_console_streams",
"with_binary_flag",
"windows_pid_is_alive",
]
+2 -1
View File
@@ -16,6 +16,7 @@ from jsonschema import Draft202012Validator
from ..scene.archive import canonical_zip_bytes, preflight_zip_bytes
from ..recording.source_mapping import canonical_source_payload
from .._internal.os_compat import with_binary_flag
from ..topology import OperationGraph, semantic_delta_to_dict, topo_delta_to_dict
from .canonical import (
DEFAULT_ARTIFACT_LIMITS,
@@ -66,7 +67,7 @@ def _schema_validate(manifest: Mapping[str, Any]) -> None:
def _stable_source_bytes(path: Path, *, max_bytes: int, error_path: str) -> bytes:
flags = os.O_RDONLY
flags = with_binary_flag(os.O_RDONLY)
if hasattr(os, "O_CLOEXEC"):
flags |= os.O_CLOEXEC
if hasattr(os, "O_NOFOLLOW"):
+3 -6
View File
@@ -10,6 +10,7 @@ from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Iterator, Mapping
from .._internal.os_compat import fsync_directory, with_binary_flag
from .canonical import (
ArtifactValidationError,
canonical_bytes,
@@ -273,11 +274,7 @@ def _atomic_write(path: Path, payload: bytes) -> None:
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary_name, path)
parent_descriptor = os.open(path.parent, os.O_RDONLY)
try:
os.fsync(parent_descriptor)
finally:
os.close(parent_descriptor)
fsync_directory(path.parent)
except BaseException:
try:
os.unlink(temporary_name)
@@ -299,7 +296,7 @@ def _state_lock(
try:
descriptor = os.open(
lock_path,
os.O_WRONLY | os.O_CREAT | os.O_EXCL,
with_binary_flag(os.O_WRONLY | os.O_CREAT | os.O_EXCL),
0o644,
)
os.close(descriptor)
+2 -1
View File
@@ -8,6 +8,7 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Iterable
from .._internal.os_compat import with_binary_flag
from ..artifacts.canonical import (
DEFAULT_ARTIFACT_LIMITS,
ArtifactLimits,
@@ -47,7 +48,7 @@ def read_stable_file(
) -> bytes:
"""Read a regular file once and reject observable concurrent mutation."""
flags = os.O_RDONLY
flags = with_binary_flag(os.O_RDONLY)
if hasattr(os, "O_CLOEXEC"):
flags |= os.O_CLOEXEC
if hasattr(os, "O_NOFOLLOW"):
+26 -16
View File
@@ -20,6 +20,11 @@ from ..artifacts.canonical import (
sha256_bytes,
validate_hash,
)
from .._internal.os_compat import (
fsync_directory,
windows_pid_is_alive,
with_binary_flag,
)
from .policy import CachePolicy
from .records import CacheRecord
@@ -92,17 +97,17 @@ class ContentAddressedStore:
@staticmethod
def _fsync_directory(path: Path) -> None:
descriptor = os.open(path, os.O_RDONLY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
fsync_directory(path)
@classmethod
def _atomic_write(cls, path: Path, payload: bytes) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.parent / f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}.tmp"
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o644)
descriptor = os.open(
temporary,
with_binary_flag(os.O_WRONLY | os.O_CREAT | os.O_EXCL),
0o644,
)
try:
with os.fdopen(descriptor, "wb", closefd=True) as stream:
stream.write(payload)
@@ -315,6 +320,9 @@ class ContentAddressedStore:
@staticmethod
def _pid_is_alive(pid: int) -> bool:
if os.name == "nt":
# os.kill(pid, 0) TerminateProcess()es its target on Windows.
return windows_pid_is_alive(pid)
try:
os.kill(pid, 0)
except ProcessLookupError:
@@ -327,11 +335,6 @@ class ContentAddressedStore:
return True
return True
@classmethod
def _can_break_stale_lock(cls, path: Path) -> bool:
owner = cls._read_lock_owner(path)
return owner is None or not cls._pid_is_alive(owner[0])
@contextmanager
def key_lock(self, namespace: str, key: str) -> Iterator[None]:
path = self.lock_path(namespace, key)
@@ -349,7 +352,7 @@ class ContentAddressedStore:
try:
descriptor = os.open(
path,
os.O_WRONLY | os.O_CREAT | os.O_EXCL,
with_binary_flag(os.O_WRONLY | os.O_CREAT | os.O_EXCL),
0o644,
)
try:
@@ -364,10 +367,17 @@ class ContentAddressedStore:
age = time.time() - path.stat().st_mtime
except FileNotFoundError:
continue
if (
age > self.policy.stale_lock_seconds
and self._can_break_stale_lock(path)
):
owner = self._read_lock_owner(path)
if owner is None:
# An unreadable owner is only weak evidence of
# abandonment; keep demanding the stale age threshold.
breakable = age > self.policy.stale_lock_seconds
else:
# A provably dead owner is positive evidence: recover at
# once instead of making every peer burn the full lock
# timeout after a crashed holder.
breakable = not self._pid_is_alive(owner[0])
if breakable:
stale = path.with_name(
path.name + f".{uuid.uuid4().hex}.stale"
)
+2 -1
View File
@@ -22,6 +22,7 @@ from .resources import (
preflight_input_archive_size,
preflight_member_compression_ratio,
)
from .._internal.os_compat import with_binary_flag
_MEMBER_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/-]{0,1023}$")
@@ -428,7 +429,7 @@ def preflight_unpacked_scene(
members: dict[str, bytes] = {}
nofollow = getattr(os, "O_NOFOLLOW", 0)
for name, (file_path, expected) in records.items():
descriptor = os.open(file_path, os.O_RDONLY | nofollow)
descriptor = os.open(file_path, with_binary_flag(os.O_RDONLY | nofollow))
try:
actual = os.fstat(descriptor)
if (
+21
View File
@@ -101,6 +101,27 @@ def test_stale_lock_is_recovered(tmp_path):
assert not lock.exists()
def test_dead_owner_lock_is_recovered_without_waiting(tmp_path):
# A crashed holder leaves a lock whose owner pid is provably dead. Peers
# recover from that positive evidence at once; only unreadable owners
# wait out the age threshold, and live owners are never broken.
from simplecadapi.artifacts.canonical import canonical_bytes
store = _store(tmp_path, lock_timeout_seconds=5.0, stale_lock_seconds=3600.0)
key = sha256_bytes(b"dead-owner")
lock = store.lock_path("part", key)
lock.parent.mkdir(parents=True, exist_ok=True)
lock.write_bytes(
canonical_bytes({"pid": 999_999_999, "token": "gone", "created_ns": "0"})
)
# Fresh mtime: the age gate alone would not open within the timeout.
with store.key_lock("part", key):
pass
assert not lock.exists()
def test_live_lock_times_out_without_overwriting_owner(tmp_path):
store = _store(tmp_path, lock_timeout_seconds=0.05, stale_lock_seconds=30.0)
key = sha256_bytes(b"live")
+183
View File
@@ -0,0 +1,183 @@
import io
import os
import subprocess
import sys
from simplecadapi._internal.os_compat import (
fsync_directory,
harden_console_streams,
windows_pid_is_alive,
with_binary_flag,
)
from simplecadapi.build.dependencies import read_stable_file
from simplecadapi.cache.store import ContentAddressedStore
def test_with_binary_flag_or_in_the_attribute_when_present(monkeypatch):
fake = 1 << 28
monkeypatch.setattr(os, "O_BINARY", fake, raising=False)
assert with_binary_flag(os.O_RDONLY) == os.O_RDONLY | fake
def test_with_binary_flag_passes_through_when_attribute_missing(monkeypatch):
monkeypatch.delattr(os, "O_BINARY", raising=False)
assert with_binary_flag(os.O_RDONLY) == os.O_RDONLY
def test_fsync_directory_is_a_noop_off_posix(tmp_path, monkeypatch):
monkeypatch.setattr(os, "name", "nt")
def _forbidden(*args, **kwargs):
raise AssertionError("directory fsync must not open files off POSIX")
monkeypatch.setattr(os, "open", _forbidden)
fsync_directory(tmp_path)
def test_fsync_directory_syncs_on_posix(tmp_path):
fsync_directory(tmp_path)
def _fake_kernel32(monkeypatch, *, last_error=87):
import ctypes
class _FakeKernel32:
def __init__(self):
self.exit_code = 259
self.closed = []
def OpenProcess(self, access, inherit, pid):
return 0 if pid == 424242 else 1
def GetExitCodeProcess(self, handle, out):
out._obj.value = self.exit_code
return 1
def CloseHandle(self, handle):
self.closed.append(handle)
kernel = _FakeKernel32()
monkeypatch.setattr(
ctypes, "WinDLL", lambda *args, **kwargs: kernel, raising=False
)
monkeypatch.setattr(
ctypes, "get_last_error", lambda: last_error, raising=False
)
return kernel
def test_windows_pid_is_alive_probes_without_killing(monkeypatch):
kernel = _fake_kernel32(monkeypatch)
assert windows_pid_is_alive(424242) is False # no such process
assert windows_pid_is_alive(1) is True # STILL_ACTIVE
kernel.exit_code = 0
assert windows_pid_is_alive(1) is False # exited
assert kernel.closed == [1, 1]
def test_windows_pid_is_alive_treats_denied_access_as_alive(monkeypatch):
_fake_kernel32(monkeypatch, last_error=5) # ERROR_ACCESS_DENIED
assert windows_pid_is_alive(424242) is True
def test_pid_is_alive_routes_to_the_windows_probe_off_posix(monkeypatch):
monkeypatch.setattr(os, "name", "nt")
_fake_kernel32(monkeypatch)
assert ContentAddressedStore._pid_is_alive(1) is True
assert ContentAddressedStore._pid_is_alive(424242) is False
def test_pid_is_alive_reports_own_and_reaped_processes():
assert ContentAddressedStore._pid_is_alive(os.getpid()) is True
child = subprocess.Popen([sys.executable, "-c", "pass"])
child.wait()
assert ContentAddressedStore._pid_is_alive(child.pid) is False
def _spy_binary_open(monkeypatch):
fake = 1 << 28
real_open = os.open
seen = []
def spy(path, flags, *args, **kwargs):
seen.append(flags)
return real_open(path, flags, *args, **kwargs)
monkeypatch.setattr(os, "open", spy)
monkeypatch.setattr(os, "O_BINARY", fake, raising=False)
return seen, fake
def test_read_stable_file_opens_in_binary_mode(tmp_path, monkeypatch):
seen, fake = _spy_binary_open(monkeypatch)
target = tmp_path / "builder.py"
target.write_bytes(b"x = 1\r\ny = 2\r\n")
payload = read_stable_file(
target, max_bytes=1024, error_path="/file_input/path"
)
assert payload == b"x = 1\r\ny = 2\r\n"
assert seen and seen[0] & fake
def test_store_atomic_write_opens_in_binary_mode(tmp_path, monkeypatch):
seen, fake = _spy_binary_open(monkeypatch)
target = tmp_path / "objects" / "aa" / "bb"
ContentAddressedStore._atomic_write(target, b"payload\nline\n")
assert target.read_bytes() == b"payload\nline\n"
assert seen and seen[0] & fake
def test_stable_source_bytes_opens_in_binary_mode(tmp_path, monkeypatch):
from simplecadapi.artifacts.feature_graph import _stable_source_bytes
seen, fake = _spy_binary_open(monkeypatch)
target = tmp_path / "source.py"
target.write_bytes(b"a = 1\n")
payload = _stable_source_bytes(target, max_bytes=1024, error_path="/source")
assert payload == b"a = 1\n"
assert seen and seen[0] & fake
def test_key_lock_opens_in_binary_mode(tmp_path, monkeypatch):
from simplecadapi.cache.policy import CachePolicy
seen, fake = _spy_binary_open(monkeypatch)
store = ContentAddressedStore(CachePolicy(root=tmp_path / "cache"))
with store.key_lock("part", "sha256:" + "0" * 64):
pass
assert seen and seen[0] & fake
def test_harden_console_streams_escapes_unencodable_diagnostics():
raw = io.BytesIO()
stream = io.TextIOWrapper(raw, encoding="ascii", errors="strict", newline="\n")
harden_console_streams((stream,))
stream.write("警告: wire fault\n")
stream.flush()
assert raw.getvalue() == b"\\u8b66\\u544a: wire fault\n"
def test_harden_console_streams_keeps_utf_streams_untouched():
stream = io.TextIOWrapper(io.BytesIO(), encoding="utf-8", errors="strict")
harden_console_streams((stream,))
assert stream.errors == "strict"
def test_harden_console_streams_tolerates_plain_and_closed_streams():
harden_console_streams((object(),))
closed = io.TextIOWrapper(io.BytesIO(), encoding="ascii")
closed.close()
harden_console_streams((closed,))