Goal (user, continuing the v13 /goal): keep taking the engine toward being (1)
practically usable across other projects, (2) research-complete, (3) the agent’s
own skill — advancing the physical-AI perception pipeline the application
vectors call for (onocollo video, evis/hillco pose & stereo depth, locomotion
terrain, grasp). Everything is local; nothing is pushed (human-gated). The full
suite stayed green at each step (2482 → 2497 passed).
The v13 stack measured space (stereo depth), terrain, objects, pose, and registration. v14 adds the missing time / motion axis and hardens the depth, terrain, and registration blocks for real (noisy, partial, tilted) input.
flow.py, tests/test_flow.py)Dense two-frame motion, so a rendered physics clip (onocollo) or a moving figure (evis/hillco body language) yields a per-pixel displacement field.
optical_flow_lk(prev, nxt, window, levels, iters, reg) — coarse-to-fine
(Gaussian-pyramid) Lucas-Kanade; recovers multi-pixel motion, Tikhonov-regularised
so flat/aperture-ambiguous regions resolve to zero flow.optical_flow_hs(prev, nxt, alpha, iters) — Horn-Schunck global smoothness
(Jacobi), a second estimator to cross-check LK.warp_by_flow(img, u, v) — forward warp (out[y,x]=img[y-v,x-u]); warping prev
by its flow reconstructs nxt — the standard consistency check.flow_magnitude / flow_angle; and imgio.colorize_flow(u,v) — Middlebury
colour wheel (direction = hue, speed = brightness; zero motion → black).nxt = ndimage.shift(prev, (v0,u0)) → recovered flow matches the
exact (u0,v0) in the interior; warp reconstructs nxt; zero motion → exact 0.stereo.py)disparity_subpixel(left, right, ...) — parabola fit through the winning cost
and its two neighbours → sub-pixel disparity (a true 5.4-px shift reads ~5.4, not
5). Sharpens metric depth for evis stereo. The integer WTA disparity_map
behaviour is unchanged (a shared internal cost volume is argmin’d exactly as before).disparity_map(..., reference='left'|'right') — the right-referenced map (via
the standard mirror identity) needed for the check below.lr_consistency(dispL, dispR, max_diff) — drops occlusions / mismatches where
the left disparity is not echoed by the right map at the matched column.terrain.py)ground_plane(grid, trim, iters) — robust trimmed least-squares fit of the
walkable plane (flat or tilted ramp), discarding the highest-residual cells
(the obstacles) each iteration. No boundary artefact.ground_surface(grid, cell, radius) — morphological grey-opening envelope, the
alternative for rough/curved (non-planar) terrain.detect_obstacles(grid, cell, clearance, ground='plane'|'opening'|array, ...) —
labels cells rising above the walkable ground; returns a mask + per-obstacle
dicts (area m², peak height, cell & world centroid via extent, bbox),
largest-first. Serves locomotion obstacle avoidance / foothold planning.registration.py)pca_align(src, dst) — principal-axis coarse alignment with sign
disambiguation (tries the proper-rotation sign combinations, keeps lowest NN
RMSE); an ICP initialiser that escapes large rotations plain ICP-from-identity
cannot.icp(..., init=(R,t), trim=frac) — seedable + Trimmed ICP (keep best
1-trim matches each iteration) for outliers / partial CAD overlap. The
existing icp(src, dst) call is unchanged.register(src, dst, trim=0.2) — one call: pca_align → trimmed ICP → (R,t,aligned,rmse).point_to_plane_icp(src, dst, ...) — aligns along the destination surface
normal (normals via pointcloud.estimate_normals), which lets a source point
slide within the tangent plane and converges tighter than point-to-point on a
surface (Low 2004). Ground truth: an ellipsoid transform recovered to rmse <1e-4.register (rmse <1e-3) where
plain ICP is >10× worse; with ~13% unmatched outlier source points, trimmed ICP
recovers the inlier transform (atol 1e-2) while vanilla ICP does not.motion.py, tests/test_motion.py)Interprets a flow field into actionable signals (flow estimates motion; motion
reads it):
frame_motion_energy(u, v) — RMS speed, one scalar per frame pair; its peaks
over a clip cue events (impact, contact, a sudden move).dominant_motion(u, v, robust=True) — robust (trimmed) affine fit of the
global/camera motion [u; v] = M·[1, x, y]; a moving object does not bias it.flow_from_model / residual_motion — remove the global motion to expose the
independent object motion.motion_segments(u, v, threshold) — label independently-moving regions
(mask + per-region dicts, largest-first); threshold sized from frame_motion_energy.motion_energy_series(frames) + detect_events(energy) — per-frame-pair motion
energy across a clip and its peaks, cueing events (contact / impact / sudden move)
for onocollo physics video.dominant_motion
recovers the exact coefficients; static background + one moving patch →
motion_segments finds exactly that patch and robust dominant_motion ignores it;
a static-then-shift sequence → detect_events fires at the transition frame.pointcloud.py, tests/test_pointcloud.py)The 3-D companion to stereo (builds the cloud) and registration (aligns it):
estimate_normals(points, k, viewpoint) — per-point surface normal by local-PCA
(smallest-eigenvalue direction of the k-NN covariance), viewpoint-orientable.
A grasp approach direction / surface orientation; enables point-to-plane error.voxel_downsample(points, voxel) — one centroid per occupied voxel; bounds ICP
cost and evens out non-uniform depth-sampling density.remove_statistical_outliers(points, k, std_ratio) / remove_radius_outliers(points,
radius, min_neighbors) — clean stray/isolated points from a noisy stereo cloud
before registration or normals (return (filtered, keep_mask)).All new functions are re-exported through api.py and the fullseye facade
(import fullseye; fullseye.optical_flow_lk / disparity_subpixel / detect_obstacles
/ register / motion_segments / estimate_normals / colorize_flow …). +26 ground-truth
tests across test_flow.py, test_stereo.py, test_terrain.py,
test_registration.py, test_motion.py, test_pointcloud.py. Commits a8fe121
(flow), e2feaf8 (stereo/terrain/registration robustness), a5b2516 (motion),
74ed047 (point cloud). Full suite 2508 passed.
examples/motion_analysis.py — two frames → optical flow → global-motion
removal → moving-region segmentation (+ colourised flow export). Recovers the
synthetic ~1.5-px global drift and finds the independently moving object.examples/grasp_pose.py — noisy, partial observed cloud → voxel_downsample →
estimate_normals → register → 6-DoF object pose (rot error ~0.04° on
synthetic). Swap in a real CAD model and a depth/stereo reconstruction.flow.track_points(prev, nxt, points) — a Lucas-Kanade sparse point tracker:
follow specific points (a marker on an evis/hillco limb, an object across an
onocollo clip) by bilinearly sampling the dense LK flow. Returns
(tracked (N,2), ok mask). Ground truth: a known shift is recovered per point.pointcloud.fpfh(points, normals, k, bins) — Fast Point Feature Histogram
(Rusu et al. 2009): a rotation-invariant 3*bins-D Darboux-angle descriptor per
point (SPFH + neighbour smoothing) for correspondence.registration.feature_register(src, dst) — FPFH descriptor matching → RANSAC
over 3-point samples (most inliers) → refit → Trimmed ICP. Registers clouds whose
global axes are ambiguous (so pca_align fails) but whose local geometry is
distinctive. Ground truth: an asymmetric two-bump surface recovers a known
transform exactly (rmse<0.02, 0° rotation error, 3 seeds). Honest limits: FPFH is
rotation-invariant only with consistently-oriented normals (estimate_normals
signs are arbitrary), so matching is approximate and RANSAC+ICP absorb the noise;
a genuinely symmetric shape stays ambiguous (registration is ill-posed).The HDevelop-style studio.py gains a Perception (v14) row: load a second
frame (B), pick a mode — optical flow / motion overlay / stereo depth /
stereo terrain — and Run to render a colourised view in the existing zoom/pan
image view (Save / hover / histogram reuse the same path). The logic lives in a
Qt-free PerceptionModel(frame_b).view(mode, frame_a) → (H,W,3) RGB, unit-tested
independently of the GUI; win._perception exposes it for headless driving.
After the v14 build a 6-agent adversarial review (one reviewer per new module) hunted
for correctness bugs the ground-truth tests missed — because those tests used contrast
1.0, few iterations, cropped borders, and only well-formed inputs. It surfaced 15
real findings, every one reproduced against the actual code before fixing (commit
ae0433d, +14 regression tests, suite 2525 passed):
reg over-damped low-contrast
[0,1] pairs (70% underestimate) → made it a fraction of the mean gradient energy
(contrast-invariant); the fixed-template iteration diverged as iters grew (225-px
runaway at 80) → added a monotonic-residual guard + out-of-frame freeze + step clamp,
bounding border spikes; fixed the pyramid prolongation factor to exactly 2.dominant_motion fitted NaN samples and collapsed to a
fake zero model → fits finite samples only, returns NaN when too few; iters=1 did
no robust trim → restructured; motion_segments measured area/mean_speed on a blurred
field → measures the true field (morphological closing to knit).lr_consistency clamped negative match columns to col 0 and
fabricated agreement on the overlap-free margin → marks them invalid.ground_plane returned a fake z=0 on an all-unobserved
map → returns NaN; min_area used round → ceil; ground_surface docstring now honest
about the opening-path ramp-border artefact (the default plane path handles ramps).pca_align/register crashed on <3 points → centroid
fallback; icp reported a one-iteration-stale rmse → recomputed for the returned pose.estimate_normals crashed on an empty cloud → returns
empty; voxel_downsample silently mis-handled voxel<=0 → validates.Second-opinion review (external model, Codex). Running Codex read-only over
flow/motion/registration surfaced 6 findings — 5 real, 1 false positive (it
claimed register crashes on 2 points, which a test disproves; Codex can’t execute
in its read-only sandbox and reasoned statically). Notably it caught a regression
the first round introduced — motion_segments’ morphological closing eroded a
moving region touching the frame border → pad by edge-replication then crop. Also:
optical_flow_hs alpha=0 → 0/0 NaN (floored the denominator); optical_flow_lk/hs
now reject a <2-px dimension with a clear error; point_to_plane_icp max_iter=0
returned rmse=inf (now recomputes the residual for the returned pose); kabsch
rejects empty inputs. +5 regression tests (commit d7be5bf). Full suite 2532 passed.
Lesson: an external model’s independent view catches blind spots — and self-introduced
regressions — that same-family review misses; verify every finding against real code
(sandbox limits make false positives normal).
ground_plane assumes the walkable ground is roughly planar within the tile;
strongly curved terrain wants ground='opening'.pca_align needs an anisotropic cloud (distinct principal extents); a
near-spherical cloud has ambiguous axes and it degrades to an arbitrary start.