* claim: #4172 * GPU: a lost device stops drawing instead of crashing (#4172) After the driver resets the GPU (Windows TDR), the first queue write to meet the lost device panicked inside wgpu-core (PendingWrites::activate unwrapping DeviceError::Lost). Nothing watched the device for its loss. - egui-wgpu: RenderState::device_lost (DeviceLost) is set by the device's lost callback. The painter draws nothing on a lost device, and before a frame's uploads it submits nothing (DeviceLost::probe), so a loss the driver reports is found by that submission instead of the first write. - Pictures (thumbnails of saves, ui.render) watch the same flag: on a lost device the capturer refuses with GpuError::DeviceLost without touching it, and the picture comes from the CPU rasterizer. Headless capturers watch their own devices. - The app, seeing the loss, closes keeping unsaved changes for recovery (as a forced quit does) and starts again; the new start says what happened. A second loss within ten minutes starts on the software renderer. The loss is noted in the crash journal.
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
//! The graphics device was lost after the window opened: the driver was reset (on Windows a TDR,
|
||||
//! "Display driver stopped responding and has recovered") or the GPU went away (#4172). A lost
|
||||
//! device can't draw again, so the window can't show anything more, not even a message. The
|
||||
//! painter stops drawing (`egui_wgpu::DeviceLost`); the app then closes, keeping unsaved changes
|
||||
//! for recovery as a forced quit does, and starts again ([`restart`]), and the new start says what
|
||||
//! happened ([`note`]). A device lost again soon after such a restart starts the next one on the
|
||||
//! software renderer, which doesn't depend on the graphics driver.
|
||||
|
||||
use crate::startup;
|
||||
|
||||
/// Set on the start after a lost device: when the device was lost (seconds since 1970).
|
||||
pub const LOST_ENV: &str = "SOLIDCRAFT_DEVICE_LOST";
|
||||
|
||||
/// A loss this recent (seconds) counts for the start after it: its note, and a second loss
|
||||
/// within it starts on the software renderer. Inherited by later starts, an older stamp is
|
||||
/// ignored.
|
||||
const RECENT_SECS: u64 = 600;
|
||||
|
||||
/// Seconds since 1970.
|
||||
pub fn now_secs() -> u64 {
|
||||
std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs())
|
||||
}
|
||||
|
||||
/// Whether `stamp` ([`LOST_ENV`]) names a loss at most [`RECENT_SECS`] before `now`.
|
||||
pub fn recent(stamp: Option<&str>, now: u64) -> bool {
|
||||
stamp.and_then(|s| s.trim().parse::<u64>().ok()).is_some_and(|at| at <= now && now - at <= RECENT_SECS)
|
||||
}
|
||||
|
||||
/// Whether the start after a lost device uses the software renderer: when this launch already
|
||||
/// followed a recent loss (the driver keeps losing it), or ran on it anyway.
|
||||
pub fn restart_on_software(after_recent_loss: bool, cpu_renderer: bool) -> bool {
|
||||
after_recent_loss || cpu_renderer
|
||||
}
|
||||
|
||||
/// What the start after a lost device tells the user, `software` when it runs on the software
|
||||
/// renderer.
|
||||
pub fn note(software: bool) -> String {
|
||||
let mut note = String::from(
|
||||
"The graphics device was reset (the graphics driver stopped responding), so SolidCraft closed and started again. Unsaved changes were kept for recovery.",
|
||||
);
|
||||
if software {
|
||||
note.push_str(" This start uses software rendering, which doesn't depend on the graphics driver.");
|
||||
}
|
||||
note
|
||||
}
|
||||
|
||||
/// Start SolidCraft again with the same arguments, the loss stamped at `lost_at`, on the software
|
||||
/// renderer when `software`.
|
||||
pub fn restart(software: bool, lost_at: u64) {
|
||||
let Ok(exe) = std::env::current_exe() else { return };
|
||||
let mut command = std::process::Command::new(exe);
|
||||
command.args(std::env::args_os().skip(1)).env(LOST_ENV, lost_at.to_string());
|
||||
if software {
|
||||
command.env(startup::RENDERER_ENV, "cpu");
|
||||
}
|
||||
#[cfg(windows)]
|
||||
{
|
||||
use std::os::windows::process::CommandExt;
|
||||
// Not tied to this process: its own process group, no console.
|
||||
command.creation_flags(0x0000_0200 | 0x0000_0008);
|
||||
}
|
||||
if let Err(e) = command.spawn() {
|
||||
log::error!("graphics: starting again after the device was lost: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
/// Only a recent, well-formed stamp counts: a later start that inherited it says nothing.
|
||||
#[test]
|
||||
fn only_a_recent_loss_counts() {
|
||||
let now = 1_000_000;
|
||||
assert!(super::recent(Some("999990"), now));
|
||||
assert!(super::recent(Some(" 999400 "), now));
|
||||
assert!(!super::recent(Some("999399"), now), "older than ten minutes");
|
||||
assert!(!super::recent(Some("1000001"), now), "from the future");
|
||||
assert!(!super::recent(Some("soon"), now));
|
||||
assert!(!super::recent(None, now));
|
||||
}
|
||||
|
||||
/// The first loss starts again on the GPU; a second one soon after, on the software renderer.
|
||||
#[test]
|
||||
fn a_second_loss_in_a_row_starts_on_the_software_renderer() {
|
||||
assert!(!super::restart_on_software(false, false));
|
||||
assert!(super::restart_on_software(true, false));
|
||||
assert!(super::restart_on_software(false, true));
|
||||
assert!(super::note(true).contains("software rendering"));
|
||||
assert!(!super::note(false).contains("software rendering"));
|
||||
assert!(super::note(false).contains("kept for recovery"));
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,7 @@
|
||||
#[path = "../../../assets/icons/app/shared/source/app_icon.rs"]
|
||||
mod app_icon;
|
||||
mod control_server;
|
||||
mod device_lost;
|
||||
#[cfg(target_os = "windows")]
|
||||
mod native_crash;
|
||||
#[cfg(target_os = "windows")]
|
||||
@@ -76,6 +77,8 @@ struct App {
|
||||
frames: u32,
|
||||
/// Started over the control channel: the window doesn't ask for the foreground.
|
||||
take_focus_unwanted: bool,
|
||||
/// The graphics device: lost, it can't draw again (`App::watch_device`, #4172).
|
||||
gpu: Option<LostWatch>,
|
||||
#[cfg(any(target_os = "windows", target_os = "linux", target_os = "macos"))]
|
||||
space_mouse: space_mouse::SpaceMouse,
|
||||
}
|
||||
@@ -107,6 +110,7 @@ impl eframe::App for App {
|
||||
}
|
||||
}
|
||||
}
|
||||
self.watch_device();
|
||||
// The design on screen is the one whose commands a crash report replays.
|
||||
self.solve.session.crash_journal = solidcraft_engine::crash::enabled();
|
||||
// The OS closing the window (Alt+F4, Cmd+Q, the window manager): ask about unsaved
|
||||
@@ -147,6 +151,9 @@ impl eframe::App for App {
|
||||
solidcraft_ui_egui::save_job::tick(&mut self.solve, true);
|
||||
solidcraft_ui_egui::recovery_ui::close(&mut self.solve);
|
||||
solidcraft_engine::crash::end_session();
|
||||
if let Some(lost_at) = self.gpu.as_ref().and_then(|g| g.lost_at) {
|
||||
device_lost::restart(self.gpu.as_ref().is_some_and(|g| g.restart_on_software), lost_at);
|
||||
}
|
||||
// An update asked to restart into the new version or to run its installer.
|
||||
if let Some((program, args)) = solidcraft_ui_egui::update::take_launch(&mut self.solve) {
|
||||
let mut command = std::process::Command::new(&program);
|
||||
@@ -166,6 +173,30 @@ impl eframe::App for App {
|
||||
}
|
||||
}
|
||||
|
||||
/// The window's graphics device, watched for its loss.
|
||||
struct LostWatch {
|
||||
lost: eframe::egui_wgpu::DeviceLost,
|
||||
/// Where the next start runs once this device is lost ([`device_lost::restart_on_software`]).
|
||||
restart_on_software: bool,
|
||||
/// When the loss was seen: the app is closing to start again.
|
||||
lost_at: Option<u64>,
|
||||
}
|
||||
|
||||
impl App {
|
||||
/// The graphics device was lost (#4172): the window can't draw again, so close, keeping
|
||||
/// unsaved changes for recovery as a forced quit does (the window can't ask about them), and
|
||||
/// start again once closed ([`device_lost::restart`]).
|
||||
fn watch_device(&mut self) {
|
||||
let Some(gpu) = self.gpu.as_mut().filter(|g| g.lost_at.is_none() && g.lost.is_lost()) else { return };
|
||||
let reason = gpu.lost.reason().unwrap_or_default();
|
||||
log::error!("graphics: the device was lost ({reason}); closing to start again");
|
||||
eprintln!("SolidCraft graphics: the device was lost ({reason}); starting again");
|
||||
solidcraft_engine::crash::phase(&format!("graphics device lost: {reason}"));
|
||||
gpu.lost_at = Some(device_lost::now_secs());
|
||||
self.solve.quit_requested = true;
|
||||
}
|
||||
}
|
||||
|
||||
fn services(cc: &eframe::CreationContext<'_>) -> Services {
|
||||
let parent = rfd::AsyncFileDialog::new().set_parent(cc);
|
||||
let open_parent = parent.clone();
|
||||
@@ -579,6 +610,11 @@ fn main() -> eframe::Result {
|
||||
notes.push(format!("The last {failed_starts} launches stopped before the window opened, so this one is in safe mode."));
|
||||
}
|
||||
let cpu_renderer = safe_mode || startup::cpu_renderer_env(std::env::var(startup::RENDERER_ENV).ok().as_deref());
|
||||
// The launch before this one closed because its graphics device was lost (`App::watch_device`).
|
||||
let after_loss = device_lost::recent(std::env::var(device_lost::LOST_ENV).ok().as_deref(), device_lost::now_secs());
|
||||
if after_loss {
|
||||
notes.push(device_lost::note(cpu_renderer));
|
||||
}
|
||||
if safe_mode {
|
||||
notes.push("Safe mode: default settings (yours are kept for the next normal start), no recovery offer, software rendering.".into());
|
||||
}
|
||||
@@ -669,6 +705,11 @@ fn main() -> eframe::Result {
|
||||
app.load_prefs(&p);
|
||||
}
|
||||
mark("app state and prefs");
|
||||
let gpu = cc.wgpu_render_state.as_ref().map(|rs| LostWatch {
|
||||
lost: rs.device_lost.clone(),
|
||||
restart_on_software: device_lost::restart_on_software(after_loss, cpu_renderer),
|
||||
lost_at: None,
|
||||
});
|
||||
if let Some(rs) = &cc.wgpu_render_state {
|
||||
eprintln!("SolidCraft graphics: {:?}, MSAA {}x (requested {}x)", rs.adapter.get_info().backend, rs.msaa_samples, samples);
|
||||
solidcraft_engine::crash::phase("viewport pipelines (shader compile)");
|
||||
@@ -746,6 +787,7 @@ fn main() -> eframe::Result {
|
||||
starting,
|
||||
frames: 0,
|
||||
take_focus_unwanted: control_port.is_some(),
|
||||
gpu,
|
||||
#[cfg(any(target_os = "windows", target_os = "linux", target_os = "macos"))]
|
||||
space_mouse,
|
||||
}))
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
//! error that names the policy.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use solidcraft_geom::Vec3;
|
||||
@@ -42,6 +43,9 @@ pub struct Capturer {
|
||||
depth: wgpu::TextureFormat,
|
||||
max_side: u32,
|
||||
state: Mutex<State>,
|
||||
/// The device was lost (#4172): set by the device's lost callback, which whoever made the
|
||||
/// device registered ([`Capturer::watching`]).
|
||||
lost: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
struct State {
|
||||
@@ -85,9 +89,33 @@ impl Capturer {
|
||||
state: Mutex::new(State { renderer, background, key: 0, access_version: 0 }),
|
||||
device,
|
||||
queue,
|
||||
lost: Arc::default(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Stop drawing once `lost` is set: the flag the device's lost callback sets (the app's
|
||||
/// window device has one, `egui_wgpu::DeviceLost`). A lost device can't draw again; its
|
||||
/// pictures fail with [`GpuError::DeviceLost`] without touching it (#4172).
|
||||
pub fn watching(mut self, lost: Arc<AtomicBool>) -> Self {
|
||||
self.lost = lost;
|
||||
self
|
||||
}
|
||||
|
||||
/// The device was lost: pictures fail with [`GpuError::DeviceLost`].
|
||||
pub fn is_lost(&self) -> bool {
|
||||
self.lost.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
/// [`Capturer::is_lost`], asking the driver first: an empty submission meets a loss the
|
||||
/// driver reports and marks it, where the picture's first upload would panic inside wgpu.
|
||||
fn probe_lost(&self) -> bool {
|
||||
if self.is_lost() {
|
||||
return true;
|
||||
}
|
||||
self.queue.submit([]);
|
||||
self.is_lost()
|
||||
}
|
||||
|
||||
/// A device of its own, without a window (native only): the adapter the `SOLIDCRAFT_GPU`
|
||||
/// policy picks ([`crate::select`]).
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
@@ -95,7 +123,8 @@ impl Capturer {
|
||||
let (adapter, device, queue) = crate::open_headless()?;
|
||||
// Errors are caught per picture (error scopes); anything else is logged, never a panic.
|
||||
device.on_uncaptured_error(Arc::new(|e: wgpu::Error| log::error!("GPU: {e}")));
|
||||
Self::new(&adapter, device, queue)
|
||||
let lost = watch_lost(&device);
|
||||
Ok(Self::new(&adapter, device, queue)?.watching(lost))
|
||||
}
|
||||
|
||||
/// As [`Capturer::headless`], drawing with the depth format `depth` ([`Capturer::with_depth`]).
|
||||
@@ -103,7 +132,8 @@ impl Capturer {
|
||||
pub fn headless_with_depth(depth: wgpu::TextureFormat) -> Result<Self> {
|
||||
let (adapter, device, queue) = crate::open_headless()?;
|
||||
device.on_uncaptured_error(Arc::new(|e: wgpu::Error| log::error!("GPU: {e}")));
|
||||
Self::with_depth(&adapter, device, queue, depth)
|
||||
let lost = watch_lost(&device);
|
||||
Ok(Self::with_depth(&adapter, device, queue, depth)?.watching(lost))
|
||||
}
|
||||
|
||||
/// As [`Capturer::headless`], without multisampling: one sample per pixel, as the viewport
|
||||
@@ -113,7 +143,8 @@ impl Capturer {
|
||||
let (adapter, device, queue) = crate::open_headless()?;
|
||||
device.on_uncaptured_error(Arc::new(|e: wgpu::Error| log::error!("GPU: {e}")));
|
||||
let depth = crate::depth_format(&device, true);
|
||||
Self::with_samples(&adapter, device, queue, depth, 1)
|
||||
let lost = watch_lost(&device);
|
||||
Ok(Self::with_samples(&adapter, device, queue, depth, 1)?.watching(lost))
|
||||
}
|
||||
|
||||
/// The depth format pictures are drawn with; float formats are drawn with reversed depth.
|
||||
@@ -142,6 +173,9 @@ impl Capturer {
|
||||
if !(1..=self.max_side).contains(&width) || !(1..=self.max_side).contains(&height) {
|
||||
return Err(GpuError::Render(format!("a picture is 1 to {} pixels a side, not {width} × {height}", self.max_side)));
|
||||
}
|
||||
if self.probe_lost() {
|
||||
return Err(GpuError::DeviceLost);
|
||||
}
|
||||
let mut state = self.state.lock().map_err(|_| GpuError::Render("an earlier picture failed on this device".into()))?;
|
||||
let state = &mut *state;
|
||||
state.key = state.key.wrapping_add(1);
|
||||
@@ -228,6 +262,10 @@ impl Capturer {
|
||||
size,
|
||||
);
|
||||
let submission = queue.submit([encoder.finish()]);
|
||||
// Waiting on a device the submission found lost would panic inside wgpu.
|
||||
if self.is_lost() {
|
||||
return Err(GpuError::DeviceLost);
|
||||
}
|
||||
if let Some(e) = caught(scopes) {
|
||||
return Err(GpuError::Render(e));
|
||||
}
|
||||
@@ -388,6 +426,19 @@ pub fn scene_geometry(scene: &Scene) -> GpuScene {
|
||||
sc
|
||||
}
|
||||
|
||||
/// A flag `device`'s lost callback sets (a device of this crate's own: a device has one lost
|
||||
/// callback, and the app's window device has egui-wgpu's).
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn watch_lost(device: &wgpu::Device) -> Arc<AtomicBool> {
|
||||
let lost = Arc::new(AtomicBool::new(false));
|
||||
let seen = lost.clone();
|
||||
device.set_device_lost_callback(move |reason, message| {
|
||||
log::error!("GPU: the device was lost ({reason:?}): {message}");
|
||||
seen.store(true, Ordering::Release);
|
||||
});
|
||||
lost
|
||||
}
|
||||
|
||||
/// Error scopes catching what a picture's GPU work may raise (popped by [`caught`]).
|
||||
fn error_scopes(device: &wgpu::Device) -> [wgpu::ErrorScopeGuard; 3] {
|
||||
[wgpu::ErrorFilter::OutOfMemory, wgpu::ErrorFilter::Internal, wgpu::ErrorFilter::Validation].map(|f| device.push_error_scope(f))
|
||||
|
||||
@@ -95,6 +95,8 @@ pub enum GpuError {
|
||||
Pipeline(String),
|
||||
#[error("not supported yet: {0}")]
|
||||
Unsupported(String),
|
||||
#[error("the GPU device was lost (the graphics driver was reset); it can't draw again")]
|
||||
DeviceLost,
|
||||
}
|
||||
|
||||
pub type Result<T> = std::result::Result<T, GpuError>;
|
||||
|
||||
+74
-11
@@ -33,13 +33,7 @@ pub fn install(rs: &egui_wgpu::RenderState, depth_bits: u8, stencil_bits: u8, sa
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
pub fn install_picture_renderer(rs: &egui_wgpu::RenderState) {
|
||||
match solidcraft_gpu::capture::Capturer::new(&rs.adapter, rs.device.clone(), rs.queue.clone()) {
|
||||
Ok(capturer) => {
|
||||
let capturer = std::sync::Arc::new(capturer);
|
||||
solidcraft_engine::capture::install(std::sync::Arc::new(move |scene, cam, w, h| {
|
||||
let rgba = capturer.rgba(scene, cam, w, h).map_err(|e| e.to_string())?;
|
||||
Ok(solidcraft_engine::capture::Picture { rgba, renderer: capturer.renderer() })
|
||||
}));
|
||||
}
|
||||
Ok(capturer) => install_capturer(capturer.watching(rs.device_lost.flag())),
|
||||
Err(e) => {
|
||||
log::warn!("pictures of the model are drawn by the CPU rasterizer: the GPU cannot draw them: {e}");
|
||||
install_cpu_picture_renderer();
|
||||
@@ -47,14 +41,45 @@ pub fn install_picture_renderer(rs: &egui_wgpu::RenderState) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Draw this process's headless pictures with `capturer`; once its device is lost (the graphics
|
||||
/// driver was reset, #4172) the CPU rasterizer draws them, so saving still writes a thumbnail.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn install_capturer(capturer: solidcraft_gpu::capture::Capturer) {
|
||||
let capturer = std::sync::Arc::new(capturer);
|
||||
solidcraft_engine::capture::install(std::sync::Arc::new(move |scene, cam, w, h| picture(&capturer, scene, cam, w, h)));
|
||||
}
|
||||
|
||||
/// `scene` drawn by `capturer`, or by the CPU rasterizer once its device is lost.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn picture(
|
||||
capturer: &solidcraft_gpu::capture::Capturer,
|
||||
scene: &solidcraft_engine::render::Scene,
|
||||
cam: &solidcraft_engine::render::Camera,
|
||||
w: u32,
|
||||
h: u32,
|
||||
) -> Result<solidcraft_engine::capture::Picture, String> {
|
||||
match capturer.rgba(scene, cam, w, h) {
|
||||
Ok(rgba) => Ok(solidcraft_engine::capture::Picture { rgba, renderer: capturer.renderer() }),
|
||||
Err(solidcraft_gpu::GpuError::DeviceLost) => Ok(cpu_picture(scene, cam, w, h)),
|
||||
Err(e) => Err(e.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// The last tier: draw this process's headless pictures with the CPU rasterizer, and every
|
||||
/// picture says so. The app uses it when its device cannot draw pictures; the web app always
|
||||
/// does, since a browser cannot wait for the GPU to hand a picture back.
|
||||
pub fn install_cpu_picture_renderer() {
|
||||
solidcraft_engine::capture::install(std::sync::Arc::new(|scene, cam, w, h| {
|
||||
let canvas = solidcraft_engine::render::render(scene, cam, w as usize, h as usize);
|
||||
Ok(solidcraft_engine::capture::Picture { rgba: canvas.rgba, renderer: "cpu" })
|
||||
}));
|
||||
solidcraft_engine::capture::install(std::sync::Arc::new(|scene, cam, w, h| Ok(cpu_picture(scene, cam, w, h))));
|
||||
}
|
||||
|
||||
fn cpu_picture(
|
||||
scene: &solidcraft_engine::render::Scene,
|
||||
cam: &solidcraft_engine::render::Camera,
|
||||
w: u32,
|
||||
h: u32,
|
||||
) -> solidcraft_engine::capture::Picture {
|
||||
let canvas = solidcraft_engine::render::render(scene, cam, w as usize, h as usize);
|
||||
solidcraft_engine::capture::Picture { rgba: canvas.rgba, renderer: "cpu" }
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -371,4 +396,42 @@ mod tests {
|
||||
assert!(chosen.contains(used), "the offscreen device runs on {used:?}, not on {chosen:?}");
|
||||
}
|
||||
}
|
||||
/// A device the driver loses (#4172): its lost callback marks it, the painter's probe sees it
|
||||
/// before the next frame's uploads, those uploads no longer reach the device (they panicked
|
||||
/// inside wgpu when they were the first to meet the loss), and pictures (thumbnails of saves)
|
||||
/// come from the CPU rasterizer. `destroy` loses the device as a driver reset does; wgpu calls
|
||||
/// the lost callback of a destroyed device once it has polled.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
#[test]
|
||||
fn a_lost_device_stops_drawing_and_pictures_fall_back_to_the_cpu() {
|
||||
use egui_wgpu::wgpu;
|
||||
let Some((adapter, device, queue)) = super::offscreen_device() else {
|
||||
eprintln!("skipping: no GPU adapter");
|
||||
return;
|
||||
};
|
||||
let lost = egui_wgpu::DeviceLost::watch(&device);
|
||||
let capturer = solidcraft_gpu::capture::Capturer::new(&adapter, device.clone(), queue.clone()).unwrap().watching(lost.flag());
|
||||
let (scene, cam) = (solidcraft_engine::render::Scene::default(), solidcraft_engine::render::Camera::default());
|
||||
assert_ne!(super::picture(&capturer, &scene, &cam, 16, 16).unwrap().renderer, "cpu");
|
||||
assert!(!lost.probe(&queue), "a working device isn't lost");
|
||||
|
||||
// egui's renderer is made with the window, before any loss (made on a lost device, its
|
||||
// uniform buffer's `create_buffer_init` panics).
|
||||
let mut renderer = egui_wgpu::Renderer::new(&device, wgpu::TextureFormat::Rgba8Unorm, Default::default());
|
||||
|
||||
device.destroy();
|
||||
let _ = device.poll(wgpu::PollType::wait_indefinitely());
|
||||
assert!(lost.is_lost() && lost.probe(&queue), "the lost callback marked the device");
|
||||
assert!(lost.reason().is_some_and(|r| r.starts_with("Destroyed")), "{:?}", lost.reason());
|
||||
|
||||
// The next frame's uploads, as the painter would make them were it not skipping: wgpu
|
||||
// knows the device is lost, so they fail quietly instead of panicking.
|
||||
let image = egui::ImageData::Color(std::sync::Arc::new(egui::ColorImage::filled([4, 4], egui::Color32::WHITE)));
|
||||
let delta = egui::epaint::ImageDelta::full(image, egui::TextureOptions::LINEAR);
|
||||
renderer.update_texture(&device, &queue, egui::TextureId::Managed(0), &delta);
|
||||
|
||||
// A save's thumbnail still comes: from the CPU rasterizer.
|
||||
let picture = super::picture(&capturer, &scene, &cam, 16, 16).unwrap();
|
||||
assert_eq!((picture.renderer, picture.rgba.len()), ("cpu", 16 * 16 * 4));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -91,6 +91,16 @@ from the Start menu after you sign in again. From 0.11.1 the setting can also li
|
||||
`graphics.json` in the settings folder, as `{"msaa": 4, "backend": "dx12"}` (`auto` is the
|
||||
order above). Safe mode ignores it and always uses the software renderer.
|
||||
|
||||
## SolidCraft restarted, saying the graphics device was reset
|
||||
|
||||
When the graphics driver stops responding, the system resets it (on Windows, "Display driver
|
||||
stopped responding and has recovered"), and the graphics device SolidCraft draws with is gone
|
||||
for good: the window can't draw anything more. SolidCraft then closes, keeps your unsaved
|
||||
changes for recovery as it does after a crash, and starts again; the new window says what
|
||||
happened and offers the changes back. If the device is reset again within ten minutes of that
|
||||
restart, the next start uses software rendering, which doesn't depend on the graphics driver.
|
||||
Updating the graphics driver from the GPU maker usually stops the resets.
|
||||
|
||||
## A design saved by 0.10 won't open in an older version
|
||||
|
||||
Designs saved by a newer SolidCraft use a newer file format. Older versions refuse them with
|
||||
|
||||
Vendored
+5
@@ -292,3 +292,8 @@ SolidCraft changes:
|
||||
and the frame is skipped with a repaint asked for when it is due. The streak and the deadline
|
||||
survive a recreate; a configure the device takes ends the streak, a recreate it asked for
|
||||
included. A recreate for a lost surface goes ahead as before.
|
||||
- `src/lib.rs`, `src/winit.rs`: the device's lost callback marks `RenderState::device_lost`
|
||||
(`DeviceLost`), and the painter draws nothing on a lost device. Before a frame's uploads it
|
||||
submits nothing (`DeviceLost::probe`), so a device the driver lost is found by that
|
||||
submission rather than by the first queue write, which panicked inside wgpu-core
|
||||
(`PendingWrites::activate`, #4172).
|
||||
|
||||
Vendored
+82
@@ -131,6 +131,65 @@ impl SurfaceConfig {
|
||||
};
|
||||
}
|
||||
|
||||
/// SolidCraft: whether a device was lost (the driver reset it, Windows' TDR among others, #4172).
|
||||
///
|
||||
/// A lost device can't be used again. wgpu only learns of the loss when a call into the driver
|
||||
/// fails: from then on every call on the device fails quietly, and the device's lost callback
|
||||
/// marks this. But a queue write that is the first call to meet the loss panics inside wgpu
|
||||
/// (`PendingWrites::activate`), so the painter first submits nothing ([`DeviceLost::probe`]):
|
||||
/// that submission meets the loss instead, and the frame's uploads are skipped.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct DeviceLost {
|
||||
lost: Arc<std::sync::atomic::AtomicBool>,
|
||||
reason: Arc<std::sync::Mutex<Option<String>>>,
|
||||
}
|
||||
|
||||
impl DeviceLost {
|
||||
/// Watch `device`: its lost callback marks the loss. A device has one lost callback, so this
|
||||
/// replaces any set before.
|
||||
pub fn watch(device: &wgpu::Device) -> Self {
|
||||
let lost = Self::default();
|
||||
let seen = lost.clone();
|
||||
device.set_device_lost_callback(move |reason, message| {
|
||||
seen.mark(&format!("{reason:?}: {message}"));
|
||||
});
|
||||
lost
|
||||
}
|
||||
|
||||
/// Mark the device lost, for `reason` (the first reason is kept).
|
||||
pub fn mark(&self, reason: &str) {
|
||||
if let Ok(mut kept) = self.reason.lock() {
|
||||
kept.get_or_insert_with(|| reason.to_owned());
|
||||
}
|
||||
self.lost.store(true, std::sync::atomic::Ordering::Release);
|
||||
}
|
||||
|
||||
/// The device was lost.
|
||||
pub fn is_lost(&self) -> bool {
|
||||
self.lost.load(std::sync::atomic::Ordering::Acquire)
|
||||
}
|
||||
|
||||
/// Why the device was lost, as wgpu said it.
|
||||
pub fn reason(&self) -> Option<String> {
|
||||
self.reason.lock().ok().and_then(|r| r.clone())
|
||||
}
|
||||
|
||||
/// The flag [`DeviceLost::is_lost`] reads, for others drawing on the same device.
|
||||
pub fn flag(&self) -> Arc<std::sync::atomic::AtomicBool> {
|
||||
self.lost.clone()
|
||||
}
|
||||
|
||||
/// Whether the device is lost, asking the driver first: an empty submission reaches the
|
||||
/// driver, so a loss it reports is marked here rather than met by the next upload.
|
||||
pub fn probe(&self, queue: &wgpu::Queue) -> bool {
|
||||
if self.is_lost() {
|
||||
return true;
|
||||
}
|
||||
queue.submit([]);
|
||||
self.is_lost()
|
||||
}
|
||||
}
|
||||
|
||||
/// Access to the render state for egui.
|
||||
#[derive(Clone)]
|
||||
pub struct RenderState {
|
||||
@@ -171,6 +230,9 @@ pub struct RenderState {
|
||||
///
|
||||
/// Update this to have the surface reconfigured on the next paint.
|
||||
pub surface_config: SurfaceConfig,
|
||||
|
||||
/// SolidCraft: the device was lost; the painter draws nothing more (#4172).
|
||||
pub device_lost: DeviceLost,
|
||||
}
|
||||
|
||||
async fn request_adapter(
|
||||
@@ -313,6 +375,8 @@ impl RenderState {
|
||||
queue,
|
||||
}) => (instance, adapter, device, queue),
|
||||
};
|
||||
// SolidCraft: a device the driver loses stops the painter, not the app (#4172).
|
||||
let device_lost = DeviceLost::watch(&device);
|
||||
|
||||
log_adapter_info(&adapter.get_info());
|
||||
|
||||
@@ -361,6 +425,7 @@ impl RenderState {
|
||||
msaa_supported,
|
||||
renderer: Arc::new(RwLock::new(renderer)),
|
||||
surface_config: config.surface,
|
||||
device_lost,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -872,3 +937,20 @@ pub fn clamp_surface_extent(width: u32, height: u32, max_side: u32) -> (u32, u32
|
||||
let scale_f32 = scale as f32;
|
||||
(fit(width), fit(height), scale_f32)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod device_lost_tests {
|
||||
/// A loss is marked once, with the first reason, and every clone (and the shared flag) sees it.
|
||||
#[test]
|
||||
fn a_marked_loss_is_seen_by_every_clone() {
|
||||
let lost = super::DeviceLost::default();
|
||||
let painter = lost.clone();
|
||||
let flag = lost.flag();
|
||||
assert!(!painter.is_lost() && lost.reason().is_none());
|
||||
lost.mark("Unknown: Device lost");
|
||||
lost.mark("Destroyed: ");
|
||||
assert!(painter.is_lost());
|
||||
assert!(flag.load(std::sync::atomic::Ordering::Acquire));
|
||||
assert_eq!(painter.reason().as_deref(), Some("Unknown: Device lost"));
|
||||
}
|
||||
}
|
||||
|
||||
Vendored
+26
@@ -643,6 +643,19 @@ impl Painter {
|
||||
// SolidCraft: one clock reading for this frame's surface retries (#1935).
|
||||
let now = web_time::Instant::now();
|
||||
|
||||
// SolidCraft: a lost device draws nothing more; the app sees `RenderState::device_lost`
|
||||
// and restarts (#4172).
|
||||
if self
|
||||
.render_state
|
||||
.as_ref()
|
||||
.is_some_and(|rs| rs.device_lost.is_lost())
|
||||
{
|
||||
// The app's frame (not this paint) acts on it: keep its frames coming.
|
||||
self.context
|
||||
.request_repaint_after_for(std::time::Duration::from_millis(250), viewport_id);
|
||||
return vsync_sec;
|
||||
}
|
||||
|
||||
// If the previous frame produced `CurrentSurfaceTexture::Lost`, the action match
|
||||
// below set `needs_recreate`. Recreate the surface now, before re-borrowing
|
||||
// `self.render_state` / `self.surfaces` for the rest of the paint.
|
||||
@@ -751,6 +764,19 @@ impl Painter {
|
||||
}
|
||||
};
|
||||
|
||||
// SolidCraft: before the uploads, an empty submission asks the driver whether the device
|
||||
// is still there. The first queue write to meet a lost device panics inside wgpu (#4172);
|
||||
// the submission meets it instead, marks it lost, and the frame is dropped unpresented.
|
||||
if render_state.device_lost.probe(&render_state.queue) {
|
||||
render_queue_guard.commands_submitted = true;
|
||||
log::error!(
|
||||
"The graphics device was lost: {}",
|
||||
render_state.device_lost.reason().unwrap_or_default()
|
||||
);
|
||||
self.context.request_repaint_of(viewport_id);
|
||||
return vsync_sec;
|
||||
}
|
||||
|
||||
// SolidCraft: acquire the drawable before preparing callbacks. Preparation can
|
||||
// cache GPU resources whose initialization is recorded into `encoder` (such as
|
||||
// the viewport's tessellated vertices). If acquisition skips this frame, dropping
|
||||
|
||||
Reference in New Issue
Block a user