Fix spurious deep contacts in box-box collision.

In the edge-edge path of the box-box collider, when line clipping yields
no points, the corner generators accept points whose projection
parameters are out of range and clamp them into the valid range. The
depth of such a point is the Euclidean distance between two unrelated
points, mixing lateral offset into penetration depth, and on the
penetrating side it is admitted with no margin check. For thin boxes
meeting edge-to-face within margin, this produced a contact with
penetration three orders of magnitude larger than the boxes' true
separation, exploding resting stacks.

No contact can penetrate deeper than the support-interval overlap along
the separating axis, which the SAT stage has already computed. Enforce
this bound on all points emitted by the edge-edge path. The bound
carries margin plus relative and size-scaled slack covering rounding
error: the depth of the deepest legitimate point is algebraically equal
to the bound, so an exact comparison would drop real contacts. The
slack is precision-dependent: in mjUSESINGLE builds the two
computations of the same overlap disagree by tens of ulps, and slack
calibrated for double precision rejects real single-precision contacts.

Differential fuzzing against the nativeccd oracle over 200k random
near-contact thin-box configurations, in both precisions: impossibly
deep contacts drop from 1018 to 4 (worst excess from 2.5x the bounding
diameter to 0.001x), with no legitimate shallow-penetration contact
lost.

PiperOrigin-RevId: 957628830
Change-Id: Iaa1f10742f91c55bf831296cba0936eb50ea09b0

src/engine/engine_collision_box.c

@@ -19,6 +19,17 @@
 #include "engine/engine_util_blas.h"
 #include "engine/engine_util_misc.h"
 
+// rounding slack for the edge-edge depth bound: relative, and absolute times the sum of
+// half-sizes; wide enough to cover rounding between two computations of the same overlap,
+// orders of magnitude below the box-scale depths of spurious clipping artifacts
+#ifdef mjUSESINGLE
+  #define mjDEPTHSLACKREL 1e-4f
+  #define mjDEPTHSLACKABS 1e-5f
+#else
+  #define mjDEPTHSLACKREL 1e-6
+  #define mjDEPTHSLACKABS 1e-12
+#endif
+
 // hard-clamp vector to range [-limit(i), +limit(i)]
 static void mju_clampVec(mjtNum* vec, const mjtNum* limit, int n) {
   for (int i = 0; i < n; i++) {
@@ -616,6 +627,7 @@ int _boxbox(const mjModel* M, const mjData* D, mjPreContact* con, int g1, int g2
          depth[mjMAXCONPAIR], pts[6][3], ppts2[4][2], pu[4][3], axi[3][3];
   mjtNum linesu[4][6], lines[4][6], clnorm[3], rnorm[3];
   mjtNum penetration, c1, c2, c3, a, b, c, d, lx, ly, hz, l, x, y, u, v, llx, lly, innorm, margin2;
+  mjtNum maxdepth;
 
   int i0, i1, i2;
   mjtNum f0, f1, f2;
@@ -1328,19 +1340,31 @@ edgeedge:
   mji_zero3(con[0].tangent);
 
 
-  for (i = 0; i < n; i++) {
-    con[i].dist = depth[i];
+  // no contact can be deeper than the support overlap along the separating axis: clipping
+  // against a grazing face can synthesize spurious points with arbitrarily large depth;
+  // the slack covers rounding error so the deepest legitimate point is never rejected
+  maxdepth = mju_max(0, penetration);
+  maxdepth += margin + mjDEPTHSLACKREL * maxdepth +
+              mjDEPTHSLACKABS * (size1[0] + size1[1] + size1[2] +
+                                 size2[0] + size2[1] + size2[2]);
+
+  for (i = 0, m = 0; i < n; i++) {
+    if (depth[i] < -maxdepth)
+      continue;
+
+    con[m].dist = depth[i];
     points[i][2] += hz;
 
     mji_mulMatVec3(tmp2, r, points[i]);
 
-    mji_add3(con[i].pos, tmp2, pos1);
+    mji_add3(con[m].pos, tmp2, pos1);
 
-    mji_copy3(con[i].normal, con[0].normal);
-    mji_zero3(con[i].tangent);
+    mji_copy3(con[m].normal, con[0].normal);
+    mji_zero3(con[m].tangent);
+    m++;
   }
 
-  return n;
+  return m;
 
 #undef rotaxis
 #undef rotmatx