Agent cursor follows the agent's own pointer input (ui.move, ui.click, ui.drag, ui.scroll) (#5162)

The agent cursor only acted out engine commands: pointer input sent over the control channel
reached the app as synthetic events without moving it, so an agent's clicks, right-clicks and
drags were invisible to a person watching. While the cursor is shown, each synthetic pointer event
now moves it to where the input lands, a press leaves a ripple, and a drag is followed step by
step. A request being acted out keeps the pointer; with the cursor hidden nothing changes.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
N3MI-DG
2026-10-23 13:42:59 +00:00
committed by GitHub
parent 5367ef4367
commit ae3b36ac88
3 changed files with 47 additions and 2 deletions
+39 -1
View File
@@ -4,7 +4,9 @@
//! then glides to each pick in the parameters (faces, edges and points projected to the screen,
//! sketch points, the plane of a new sketch), and only then does the command run. The reply
//! waits for the animation, so an agent's timing stays honest. With "follow camera" on, a new
//! sketch turns the view to face it and finishing the sketch turns back.
//! sketch turns the view to face it and finishing the sketch turns back. Pointer input the agent
//! sends itself (`ui.move`, `ui.click`, `ui.drag`, `ui.scroll`) moves the pointer too, as each
//! event reaches the app, and a press leaves a ripple.
//!
//! The show is bounded, whatever the request's size: besides the toolbar button the pointer
//! visits at most [`MAX_PICK_STOPS`] picks (a long point list is summarised by its first, middle
@@ -168,6 +170,19 @@ impl AgentCursor {
self.job.as_ref().filter(|j| j.stops.is_empty()).and_then(|j| j.ahead.as_ref()).map(|a| (a.started, a.label.as_str()))
}
/// Follow the pointer input an agent sends itself (`ui.move`, `ui.click`, `ui.drag`,
/// `ui.scroll`), as it reaches the app: the pointer goes where the input lands, and a press
/// leaves a ripple. A request being acted out keeps the pointer meanwhile.
pub(crate) fn follow_input(&mut self, at: Pos2, press: bool, now: f64) {
if self.job.is_some() {
return;
}
self.pos = Some(at);
if press {
self.ripples.push((at, now));
}
}
/// Stop the request the pointer waits on: answered as stopped, nothing of it applied.
pub(crate) fn stop(&mut self) -> Option<(Sender<ControlResponse>, String)> {
self.computing()?;
@@ -564,6 +579,29 @@ mod tests {
assert!(frames <= 3, "{frames}");
}
#[test]
fn the_pointer_follows_the_agents_own_mouse_input() {
let mut h = crate::scenario::Harness::new();
// Hidden: control-channel input leaves the pointer alone.
assert_eq!(h.call("ui.agent_cursor", json!({"show": false}))["ok"], true);
h.call("ui.move", json!({"x": 300, "y": 300}));
h.frames(2);
assert_eq!(h.app.agent.pos, None);
assert_eq!(h.call("ui.agent_cursor", json!({"show": true}))["ok"], true);
h.call("ui.move", json!({"x": 420, "y": 380}));
h.frames(2);
assert_eq!(h.app.agent.pos, Some(pos2(420.0, 380.0)), "a move moves it");
h.call("ui.click", json!({"x": 500, "y": 400, "button": "right"}));
h.frames(4);
assert_eq!(h.app.agent.pos, Some(pos2(500.0, 400.0)), "a click lands where it clicked");
assert!(h.app.agent.ripples.iter().any(|(at, _)| *at == pos2(500.0, 400.0)), "and leaves a ripple");
// A drag is followed (one step a frame) to where it is released.
h.call("ui.drag", json!({"x0": 600, "y0": 300, "x1": 700, "y1": 360, "steps": 4}));
h.frames(8);
assert_eq!(h.app.agent.pos, Some(pos2(700.0, 360.0)));
}
#[test]
fn speeds_read_old_and_new_settings() {
// Saved settings from before #79: slow and normal are 1×, instant stays.
+7
View File
@@ -1549,7 +1549,14 @@ impl SolveApp {
)
})
.map_or(self.synthetic.len(), |i| i + 1);
let follow = self.ui.agent_cursor.show;
for event in self.synthetic.drain(..n) {
// The agent cursor shows the agent's own pointer input too, not only its commands.
match &event {
egui::Event::PointerMoved(at) if follow => self.agent.follow_input(*at, false, self.now),
egui::Event::PointerButton { pos, pressed, .. } if follow => self.agent.follow_input(*pos, *pressed, self.now),
_ => {}
}
match &event {
egui::Event::ModifiersChanged(modifiers) => self.synthetic_mods = *modifiers,
egui::Event::PointerButton { pos, modifiers, pressed, .. } => {