GPU path tracer: don't rerun a setup that crashed in the driver (#9236) (#9666)

* claim: #9236

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* GPU path tracer: don't rerun a setup that crashed in the driver (#9236)

A driver that takes the process down while it compiles the path tracer
(RADV on an RX 7600) is beyond error scopes and unwind guards. A running
setup now keeps gpu-path-setup.json (its adapter key) in the settings
folder, removed when the setup answers or the app closes normally. A
launch that finds it for the same adapter leaves the setup failed with an
explanation and a Try again button; CPU ray tracing stays available, and
a driver update changes the key and so tries again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
PaddleStroke
2026-11-04 21:54:45 +00:00
committed by GitHub
parent 2fdcc5ae49
commit c4a63a1d43
8 changed files with 321 additions and 6 deletions
+2
View File
@@ -303,6 +303,8 @@ impl eframe::App for App {
log::warn!("a picture was still being drawn as the app closed");
}
solidcraft_ui_egui::recovery_ui::close(&mut self.solve);
// A GPU path tracer setup still running did not stop the app (#9236).
solidcraft_ui_egui::render_workspace::close(&mut self.solve);
solidcraft_ui_egui::open_elsewhere::close(&mut self.solve);
// Before a restart (`Drop for App`) starts the next instance, which would otherwise get
// nothing from the 3D mouse driver (`solidcraft_3dx_client::stop`).
@@ -0,0 +1,34 @@
{
"en": {
"render.gpu_path_crashed": "SolidCraft closed while setting it up on this graphics adapter last time, most likely inside the graphics driver, so it is not set up again by itself. Choose CPU ray tracing instead, or try again (a driver update also tries again).",
"render.gpu_path_retry": "Try again"
},
"ar": {
"render.gpu_path_crashed": "أُغلق SolidCraft أثناء إعداده على محوّل الرسوميات هذا في المرة السابقة، على الأرجح داخل برنامج تشغيل الرسوميات، لذا لن يُعاد إعداده تلقائيًا. استخدم تتبع الأشعة على المعالج المركزي بدلًا منه، أو حاول مرة أخرى (يعيد تحديث برنامج التشغيل المحاولة أيضًا).",
"render.gpu_path_retry": "حاول مرة أخرى"
},
"de": {
"render.gpu_path_crashed": "SolidCraft wurde beim letzten Einrichten auf diesem Grafikadapter beendet, wahrscheinlich im Grafiktreiber, und richtet ihn deshalb nicht von selbst erneut ein. Verwenden Sie stattdessen Raytracing auf der CPU oder versuchen Sie es erneut (nach einem Treiberupdate wird es ebenfalls erneut versucht).",
"render.gpu_path_retry": "Erneut versuchen"
},
"fr": {
"render.gpu_path_crashed": "SolidCraft s’est fermé pendant sa dernière préparation sur cet adaptateur graphique, très probablement dans le pilote graphique ; il n’est donc plus préparé de lui-même. Utilisez le lancer de rayons sur le CPU, ou réessayez (une mise à jour du pilote réessaie aussi).",
"render.gpu_path_retry": "Réessayer"
},
"he": {
"render.gpu_path_crashed": "SolidCraft נסגר בזמן ההכנה על מתאם גרפי זה בפעם הקודמת, ככל הנראה בתוך מנהל ההתקן הגרפי, ולכן ההכנה לא תתבצע שוב מעצמה. יש להשתמש במעקב קרניים על המעבד במקום זאת, או לנסות שוב (עדכון מנהל ההתקן גם מנסה שוב).",
"render.gpu_path_retry": "לנסות שוב"
},
"ja": {
"render.gpu_path_crashed": "前回このグラフィックアダプターでの準備中に SolidCraft が終了しました。グラフィックドライバー内で終了した可能性が高いため、自動では再度準備しません。代わりに CPU レイトレーシングを使うか、もう一度試してください(ドライバーを更新した場合も再試行されます)。",
"render.gpu_path_retry": "もう一度試す"
},
"vi": {
"render.gpu_path_crashed": "Lần trước SolidCraft đã đóng khi đang thiết lập nó trên bộ điều hợp đồ họa này, nhiều khả năng là trong trình điều khiển đồ họa, nên nó sẽ không tự thiết lập lại. Hãy dùng dò tia trên CPU, hoặc thử lại (cập nhật trình điều khiển cũng sẽ thử lại).",
"render.gpu_path_retry": "Thử lại"
},
"zh": {
"render.gpu_path_crashed": "上次在此图形适配器上设置时 SolidCraft 关闭了,很可能是在图形驱动程序内部,因此不会再自动设置。请改用 CPU 光线追踪,或重试(更新驱动程序后也会重试)。",
"render.gpu_path_retry": "重试"
}
}
+3
View File
@@ -1036,6 +1036,9 @@ impl SolveApp {
{
let adapter = rs.adapter.clone();
let key = render_workspace::adapter_key(&adapter.get_info());
// Next to the graphics settings: a setup that stops the app is not run again by itself.
let marker = self.services.graphics_path.as_deref().map(|p| std::path::Path::new(p).with_file_name(render_workspace::SETUP_MARKER));
self.render_workspace.watch_setup(marker);
self.render_workspace.defer_path_host(key, move || solidcraft_gpu::path::host_on_adapter(&adapter).map_err(|e| e.to_string()));
}
render_workspace::sync_backend(self);
+143 -5
View File
@@ -134,6 +134,13 @@ pub struct State {
pub setup_seconds: std::collections::BTreeMap<String, f64>,
/// The viewport adapter the current setup runs on.
pub setup_key: Option<String>,
/// The file that says a setup is running ([`SETUP_MARKER`] in the settings folder); `None`:
/// nowhere to keep it (browser, tests), so a setup that crashes is simply run again.
#[cfg(not(target_arch = "wasm32"))]
setup_marker: Option<std::path::PathBuf>,
/// The adapter whose setup was still running when the app last stopped (#9236).
#[cfg(not(target_arch = "wasm32"))]
crashed_setup: Option<String>,
pub geometry_budget: u64,
/// A GPU error the preview's last geometry upload caught.
pub upload_error: crate::gpu::UploadError,
@@ -471,6 +478,10 @@ impl Default for State {
pending_setup: None,
setup_seconds: Default::default(),
setup_key: None,
#[cfg(not(target_arch = "wasm32"))]
setup_marker: None,
#[cfg(not(target_arch = "wasm32"))]
crashed_setup: None,
geometry_budget: WEB_GEOMETRY_BYTES,
upload_error: Arc::new(Mutex::new(None)),
image_host: None,
@@ -507,13 +518,131 @@ impl State {
}
}
/// Keep the running-setup marker at `path`, and read what the last launch left there: a
/// marker still present means the app stopped during that setup. Once per launch.
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn watch_setup(&mut self, path: Option<std::path::PathBuf>) {
if self.setup_marker.is_some() {
return;
}
self.crashed_setup = path.as_deref().and_then(marker_adapter);
if let Some(key) = &self.crashed_setup {
log::warn!("the last GPU path tracer setup on {key} did not finish: the app stopped during it");
}
self.setup_marker = path;
}
/// Whether the deferred setup waits for the user because it stopped the app last time.
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn setup_crashed(&self) -> bool {
self.crashed_setup.is_some() && self.crashed_setup == self.setup_key && self.pending_setup.is_some()
}
#[cfg(not(target_arch = "wasm32"))]
fn start_deferred_path_host(&mut self) {
if matches!(self.path_host, PathHost::NotStarted)
&& let Some(create) = self.pending_setup.take()
{
self.path_host.start(create);
if !matches!(self.path_host, PathHost::NotStarted) || self.pending_setup.is_none() {
return;
}
// A driver that takes the process down while it compiles (#9236) is beyond any error
// scope or unwind guard; the marker is how the next launch learns of it.
if self.setup_crashed() {
self.path_host = PathHost::failed(crate::i18n::tr("render.gpu_path_crashed").into());
return;
}
let Some(create) = self.pending_setup.take() else { return };
if let (Some(path), Some(key)) = (&self.setup_marker, &self.setup_key) {
write_marker(path, key);
}
self.path_host.start(create);
if !matches!(self.path_host, PathHost::Preparing(..)) {
self.clear_setup_marker();
}
}
/// Run a setup that stopped the app last time again, at the user's request.
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn retry_crashed_setup(&mut self) {
if self.setup_crashed() && matches!(self.path_host, PathHost::Failed(_)) {
self.crashed_setup = None;
self.path_host = PathHost::NotStarted;
self.start_deferred_path_host();
}
}
/// Take the worker's answer if it has come: either answer means the process survived it.
fn poll_setup(&mut self) -> Option<std::time::Duration> {
let preparing = matches!(self.path_host, PathHost::Preparing(..));
let took = self.path_host.poll();
if preparing && !matches!(self.path_host, PathHost::Preparing(..)) {
self.clear_setup_marker();
}
took
}
#[cfg(target_arch = "wasm32")]
fn clear_setup_marker(&self) {}
#[cfg(not(target_arch = "wasm32"))]
fn clear_setup_marker(&self) {
if let Some(path) = &self.setup_marker
&& let Err(e) = std::fs::remove_file(path)
&& e.kind() != std::io::ErrorKind::NotFound
{
log::warn!("could not remove {}: {e}", path.display());
}
}
}
/// The marker file a running path tracer setup keeps in the settings folder.
pub const SETUP_MARKER: &str = "gpu-path-setup.json";
/// A marker is a few dozen bytes; anything longer is not one.
#[cfg(not(target_arch = "wasm32"))]
const MAX_MARKER_BYTES: u64 = 64 * 1024;
/// The adapter a marker names, if `path` holds a readable one.
#[cfg(not(target_arch = "wasm32"))]
fn marker_adapter(path: &std::path::Path) -> Option<String> {
use std::io::Read;
let mut text = String::new();
std::fs::File::open(path).ok()?.take(MAX_MARKER_BYTES + 1).read_to_string(&mut text).ok()?;
if text.len() as u64 > MAX_MARKER_BYTES {
return None;
}
let value: serde_json::Value = serde_json::from_str(&text).ok()?;
value.get("adapter")?.as_str().filter(|key| !key.is_empty()).map(str::to_owned)
}
#[cfg(not(target_arch = "wasm32"))]
fn write_marker(path: &std::path::Path, key: &str) {
if let Some(dir) = path.parent()
&& let Err(e) = std::fs::create_dir_all(dir)
{
log::warn!("could not create {}: {e}", dir.display());
}
if let Err(e) = std::fs::write(path, json!({"adapter": key}).to_string()) {
log::warn!("could not write {}: {e}", path.display());
}
}
/// Under the failure message: when the setup stopped the app last time (#9236), the button
/// that runs it again. Nothing else runs it.
pub(crate) fn retry_crashed_setup(app: &mut SolveApp, ui: &mut egui::Ui) {
#[cfg(not(target_arch = "wasm32"))]
if app.render_workspace.setup_crashed() {
let response = crate::ui_kit::button(ui, app.tr("render.gpu_path_retry"), crate::ui_kit::Button::Normal);
crate::scenario::publish_target("render:gpu-path-retry", response.rect);
if response.clicked() {
app.render_workspace.retry_crashed_setup();
}
}
#[cfg(target_arch = "wasm32")]
let _ = (app, ui);
}
/// The app is closing normally: a setup still running did not stop it.
pub fn close(app: &mut SolveApp) {
if matches!(app.render_workspace.path_host, PathHost::Preparing(..)) {
app.render_workspace.clear_setup_marker();
}
}
@@ -593,7 +722,7 @@ pub fn sync_backend(app: &mut SolveApp) {
if wants_path_host(app) {
app.render_workspace.start_deferred_path_host();
}
if let Some(took) = app.render_workspace.path_host.poll() {
if let Some(took) = app.render_workspace.poll_setup() {
record_setup(&mut app.render_workspace, took);
}
if let PathHost::Ready(host) = &app.render_workspace.path_host
@@ -661,7 +790,15 @@ pub fn reset_document(app: &mut SolveApp) {
let setup_seconds = std::mem::take(&mut app.render_workspace.setup_seconds);
let setup_key = app.render_workspace.setup_key.take();
let pending_setup = app.render_workspace.pending_setup.take();
#[cfg(not(target_arch = "wasm32"))]
let (setup_marker, crashed_setup) = (app.render_workspace.setup_marker.take(), app.render_workspace.crashed_setup.take());
app.render_workspace = State { host, image_host, path_host, setup_seconds, setup_key, pending_setup, geometry_budget, ..Default::default() };
#[cfg(not(target_arch = "wasm32"))]
{
// The running-setup marker is the application's, like the setup itself.
app.render_workspace.setup_marker = setup_marker;
app.render_workspace.crashed_setup = crashed_setup;
}
if active(app) {
app.ui.tab = "MODEL".into();
}
@@ -1115,6 +1252,7 @@ fn image_edit_controls(app: &mut SolveApp, ui: &mut egui::Ui) {
PathHost::Preparing(_, started) => setup_progress(ui, app, *started),
PathHost::Failed(error) => {
ui.add(egui::Label::new(format!("{}: {error}", app.tr("render.gpu_path_failed"))).wrap());
retry_crashed_setup(app, ui);
}
_ => {}
}
@@ -1218,11 +1218,12 @@ fn reveal_section(app: &mut SolveApp, ui: &mut egui::Ui, section: &'static str)
}
}
fn gpu_status(app: &SolveApp, ui: &mut egui::Ui) {
fn gpu_status(app: &mut SolveApp, ui: &mut egui::Ui) {
match &app.render_workspace.path_host {
super::PathHost::Preparing(_, started) => super::setup_progress(ui, app, *started),
super::PathHost::Failed(error) => {
ui_kit::text(ui, &format!("{}: {error}", app.tr("render.gpu_path_failed")), ui_kit::Text::Error("render.gpu_path_failed"));
super::retry_crashed_setup(app, ui);
}
super::PathHost::NotStarted if app.render_workspace.view.method == ImageMethod::GpuRaytrace => {
ui_kit::text(ui, app.tr("render.gpu_unavailable"), ui_kit::Text::Note("render.gpu_unavailable"));
@@ -1131,6 +1131,128 @@ mod async_path_host {
assert!(h.widget(&format!("GPU path tracer unavailable: {message}")).is_some());
assert!(image_host(&h.app, ImageMethod::GpuRaytrace).is_err());
}
fn marker_adapter_in(path: &std::path::Path) -> Option<String> {
let text = std::fs::read_to_string(path).ok()?;
serde_json::from_str::<serde_json::Value>(&text).ok()?["adapter"].as_str().map(str::to_owned)
}
/// A driver that takes the process down while it compiles the path tracer (#9236) can't be
/// caught: the marker it leaves keeps the next launch from running that setup by itself.
#[test]
fn a_setup_that_stopped_the_app_is_not_run_again_until_asked() {
use std::sync::atomic::{AtomicUsize, Ordering};
let mut h = panel_fixture();
let marker = h.dir.join(SETUP_MARKER);
let key = "AMD Radeon RX 7600 (RADV NAVI33) | Vulkan | radv Mesa 25.2.8";
std::fs::write(&marker, json!({"adapter": key}).to_string()).unwrap();
h.app.render_workspace.watch_setup(Some(marker.clone()));
let calls = Arc::new(AtomicUsize::new(0));
let observed = calls.clone();
h.app.render_workspace.defer_path_host(key.into(), move || {
observed.fetch_add(1, Ordering::SeqCst);
job::ImageHost::with_executor(Arc::new(TestExecutor)).map_err(|e| e.to_string())
});
h.app.render_workspace.view.method = ImageMethod::GpuRaytrace;
sync_backend(&mut h.app);
h.frames(5);
assert_eq!(calls.load(Ordering::SeqCst), 0, "the setup that stopped the app must not run by itself");
let PathHost::Failed(message) = &h.app.render_workspace.path_host else { panic!("expected the crash to be reported") };
assert_eq!(message, h.app.tr("render.gpu_path_crashed"));
assert!(h.widget(&format!("GPU path tracer unavailable: {message}")).is_some());
assert!(image_host(&h.app, ImageMethod::GpuRaytrace).is_err());
// The other image methods stay available.
assert!(h.app.session.render_job_host().is_some());
assert_eq!(marker_adapter_in(&marker).as_deref(), Some(key), "the marker stays until the setup answers");
// Opening another design keeps both the crash and the offer to try again.
crate::documents::adopt(&mut h.app, Session::default());
enter(&mut h.app, true).unwrap();
inspector::select_editor(&mut h.app, inspector::Editor::Output);
h.app.render_workspace.view.method = ImageMethod::GpuRaytrace;
h.frames(3);
assert_eq!(calls.load(Ordering::SeqCst), 0);
assert!(h.app.render_workspace.setup_crashed());
let retry = h.app.tr("render.gpu_path_retry").to_string();
assert!(h.widget(&retry).is_some(), "the panel offers to try again");
h.step(&json!({"widget": retry})).unwrap();
finish_setup(&mut h);
assert_eq!(calls.load(Ordering::SeqCst), 1);
assert!(matches!(h.app.render_workspace.path_host, PathHost::Ready(_)));
assert!(!marker.exists(), "a setup that answered removes its marker");
assert!(!h.app.render_workspace.setup_crashed());
}
#[test]
fn a_running_setup_keeps_a_marker_until_it_answers() {
for answer in [true, false] {
let mut h = panel_fixture();
let marker = h.dir.join("settings").join(SETUP_MARKER);
h.app.render_workspace.watch_setup(Some(marker.clone()));
let (release, wait) = mpsc::channel::<()>();
h.app.render_workspace.defer_path_host("Test GPU | Vulkan | 1.0".into(), move || {
wait.recv_timeout(Duration::from_secs(10)).unwrap();
if answer { job::ImageHost::with_executor(Arc::new(TestExecutor)).map_err(|e| e.to_string()) } else { Err("refused".into()) }
});
// Raster does not ask for the setup (#9041), so nothing runs and no marker is kept.
sync_backend(&mut h.app);
assert!(matches!(h.app.render_workspace.path_host, PathHost::NotStarted));
assert!(!marker.exists());
h.app.render_workspace.view.method = ImageMethod::GpuRaytrace;
sync_backend(&mut h.app);
assert!(matches!(h.app.render_workspace.path_host, PathHost::Preparing(..)));
assert_eq!(marker_adapter_in(&marker).as_deref(), Some("Test GPU | Vulkan | 1.0"));
release.send(()).unwrap();
finish_setup(&mut h);
assert_eq!(matches!(h.app.render_workspace.path_host, PathHost::Ready(_)), answer);
assert!(!marker.exists(), "answer {answer}: the process survived, so the marker goes");
}
}
#[test]
fn closing_during_setup_is_not_a_crash() {
let mut h = panel_fixture();
let marker = h.dir.join(SETUP_MARKER);
h.app.render_workspace.watch_setup(Some(marker.clone()));
let (release, wait) = mpsc::channel::<()>();
h.app.render_workspace.defer_path_host("Test GPU | Vulkan | 1.0".into(), move || {
let _ = wait.recv_timeout(Duration::from_secs(10));
Err("closed".into())
});
h.app.render_workspace.view.method = ImageMethod::GpuRaytrace;
sync_backend(&mut h.app);
assert!(marker.exists());
close(&mut h.app);
assert!(!marker.exists());
release.send(()).unwrap();
}
#[test]
fn markers_for_other_adapters_or_unreadable_ones_never_block_the_setup() {
let other = || json!({"adapter": "Another GPU | Vulkan | 1.0"}).to_string().into_bytes();
let hostile: [Vec<u8>; 6] = [
other(),
b"\xff\xfe not text".to_vec(),
b"{\"adapter\": 42}".to_vec(),
b"{\"adapter\": \"\"}".to_vec(),
b"[\"Test GPU | Vulkan | 1.0\"]".to_vec(),
format!("{{\"adapter\": \"Test GPU | Vulkan | 1.0\", \"pad\": \"{}\"}}", "x".repeat(70_000)).into_bytes(),
];
for bytes in hostile {
let mut h = panel_fixture();
let marker = h.dir.join(SETUP_MARKER);
std::fs::write(&marker, &bytes).unwrap();
h.app.render_workspace.watch_setup(Some(marker.clone()));
h.app.render_workspace.defer_path_host("Test GPU | Vulkan | 1.0".into(), || {
job::ImageHost::with_executor(Arc::new(TestExecutor)).map_err(|e| e.to_string())
});
h.app.render_workspace.view.method = ImageMethod::GpuRaytrace;
sync_backend(&mut h.app);
finish_setup(&mut h);
assert!(matches!(h.app.render_workspace.path_host, PathHost::Ready(_)), "{:?}", String::from_utf8_lossy(&bytes[..bytes.len().min(40)]));
assert!(!marker.exists());
}
}
}
#[cfg(not(target_arch = "wasm32"))]
+10
View File
@@ -170,6 +170,16 @@ depend on the graphics driver. Updating the graphics driver from the GPU maker u
resets. Laptops that lose the device on waking from sleep (often with an NVIDIA GPU that doesn't
drive the screen) can keep SolidCraft on the integrated GPU instead: Preferences › Graphics.
## SolidCraft closed while the GPU path tracer was being set up
Choosing **GPU path tracing** in the Render workspace first has the graphics driver compile the
path tracer, which can take minutes. A few drivers crash while compiling it, and the crash takes
SolidCraft with it; nothing inside SolidCraft can catch a crash inside the driver. SolidCraft
remembers that setup (`gpu-path-setup.json` in the settings folder) and doesn't run it again by
itself on that graphics adapter and driver: the Render workspace says what happened and offers
**Try again**. CPU ray tracing still works. A driver update also tries the setup again, and so
does deleting that file.
## A design saved by 0.10 won't open in an older version
Designs saved by a newer SolidCraft use a newer file format. Older versions refuse them with
+5
View File
@@ -17,6 +17,11 @@ the pending/ready/failed host, and failures are not retried on each viewport ins
It opens its independent job device on the viewport's selected adapter,
so software fallback and safe mode do not rediscover a failing hardware driver.
An incompatible explicit GPU policy leaves progressive rendering unavailable there.
A driver that crashes the process while it compiles the path pipeline (#9236) is beyond error
scopes and unwind guards, so a running setup keeps `gpu-path-setup.json` (its adapter key) in the
settings folder and removes it when the setup answers or the app closes normally. A launch that
finds it for the same adapter key leaves the setup failed with that explanation until the user
chooses **Try again**; a driver update changes the key and so tries again.
The path tracer runs on Vulkan, or on Metal on macOS (#3217). A viewport on D3D12 is reopened on the
same GPU through Vulkan. Where there is no Vulkan or Metal driver (Windows over
Remote Desktop or on the WARP software adapter, older drivers), the path executor reports