CopyQuad: Quadrupedal Skill Transfer from a Single Side-View Video Without a Depth Model

Preprint · under review

Mohamad H. Danesh^1,2 Raina Hu¹ Mengxue Xu¹ Hsiu-Chin Lin^1,2

Abstract

Video-to-robot pipelines for quadrupedal imitation insert a learned lifting stage, either a monocular depth model or a per-clip 3D reconstruction, to recover the one axis a side-view camera cannot see. We present CopyQuad, which omits that stage and pins the missing axis to the robot's own stance width instead. The setting is side-view footage of sagittally dominated motion, where limb motion is almost entirely in the image plane. A frozen 2D keypoint detector and a fixed analytic retargeting then turn a single uncalibrated clip into a robot reference whose foot targets are reachable by construction, so no per-clip trajectory optimization is needed. One phase-indexed reward structure, whose only per-behavior freedom is three scalars from a shared sweep, learns a dog walk, a horse walk, a horse trot and two non-periodic gestures, all deployed zero-shot on hardware. We then measure what omitting the axis costs. In a training-free rigidity test, a video depth model varies the distance between the animal's hips by 40 to 107% of its own mean and a single-image metric model by 22 to 58%, though that distance cannot move. Motion capture puts the true lateral paw travel of a dog walk at 4.5 cm peak to peak, a mean deviation near 1.4 cm against the 3.5 cm of lateral foot error our policies already leave, so the discarded axis sits below the pipeline's resolution. Depth repaired to feasibility never beats the constant, and the unrepaired estimates put the robot on the ground.

CopyQuad skips the hardest step in video-to-robot imitation: estimating depth. A side-view camera of a walking or trotting animal already shows almost everything a robot's retargeter needs, fore-aft motion and height. It hides only the sideways axis across the body, the one a depth model is normally called in to recover. CopyQuad doesn't recover it. It pins that axis to the robot's own stance width and measures what that costs, instead of assuming a depth model would do better.

5animal behaviors, one clip each

no depth modelthe missing axis is pinned, not estimated

2 speciesdog and horse, one shared pipeline

zero-shotsim to hardware, no fine-tuning

How CopyQuad works

A side-view camera is a weak-perspective projection of the sagittal plane: it reads fore-aft position and height cleanly, and hides the lateral axis across the animal's body almost entirely. Prior pipelines add a lifting stage, a monocular depth model or a per-clip 3D reconstruction, to recover that missing axis. CopyQuad skips it. A quadruped's lateral limb motion is small to begin with, so CopyQuad just pins that coordinate to the robot's own stance width and lets a fixed analytic pipeline do the rest.

CopyQuad pipeline: monocular video to 2D keypoints to 3D keypoints to robot reference motion to a trained policy — A frozen 2D keypoint detector reads the animal's pose from the video. Sagittal-plane lifting pins the hidden lateral axis to the robot's stance width. Kinematic retargeting turns that into a robot-reachable reference, and a phase-indexed policy is trained in simulation to track it.

Sagittal lifting, pinned laterally

pₖ = (sₓuₖ, y_nom, −s_z vₖ)

Each 2D keypoint gives fore-aft and vertical pixel position. The lateral coordinate is set to a fixed value read off the robot's own morphology, ±half its stance width, rather than estimated. Every foot target then lies on a track the robot can already reach, on every clip, with no per-clip repair.

Kinematic retargeting & root recovery

v̂ = d̄ × f̂

Damped least-squares IK turns the lifted keypoints into joint angles, holding the base fixed. Forward translation isn't captured by IK alone, so it's recovered separately from each foot's fore-aft sweep and the gait's footfall frequency, giving a straight-line base trajectory the policy can track.

Phase-indexed imitation policy

rₜ = wⱼrⱼ + w_f r_f + wᵣrᵣ − wₐrₐ

One DeepMimic-style policy per behavior tracks the reference at its current phase, rewarded for matching joint angles, foot positions and root motion while penalizing jerky actions. The only thing that changes between behaviors is three reward scalars, picked from one shared grid.

The reference quantities: stance tracks, robot height, foot sweep, and the detector keypoints used to scale the animal to the robot — The reference quantities. Left: the stance tracks the lateral axis is pinned to. Middle: robot height and the foot sweep used for root recovery. Right: the detector keypoints read directly from pixels, and the spans matched between the animal and the robot to scale everything to the right size.

Nothing in the video-to-reference chain is trained or fit per clip. The 2D keypoint detector runs frozen and zero-shot, the lifting and retargeting are fixed analytic rules applied identically to every clip, and the only learned component is the control policy itself, trained entirely in simulation. Of 24 candidate internet clips screened against four simple assumptions (a roughly static camera, travel roughly perpendicular to it, steady depth, and steady-state motion), 14 passed, close to three in five with no curation beyond the screen itself.

Results

Five behaviors, in simulation and on hardware

Using one pipeline and one reward structure, CopyQuad learns a dog walk, a horse walk, a horse trot, a dog bow and a dog give-paw, each from a single clip, and transfers every one of them to a physical ANYmal-D zero-shot. Agreement with the reference is coarse by design: the per-joint error concentrates in the sagittal hip and knee, the joints actually driven from the image, and not in the pinned hip abduction. The claim is gait-structure transfer, not pose-level fidelity, and the keyframes below show that structure surviving the jump from video to simulation to hardware intact.

Behavior	Dog walk	Horse walk	Horse trot	Dog bow	Dog give-paw
Behavior	clip fps / length (s)
Source clip	30 / 21.4	25 / 21	30 / 36	50 / 7	30 / 8
Sim: ep (cm) ↓	13.7±1.8	15.0±2.6	15.5±2.0	7.4±0.4	11.8±6.8
Sim: survival (s, cap 20)	19.9±0.2	19.5±0.7	19.6±0.5	20.0±0.0	15.1±2.9
Hardware: eq (deg) ↓	16.8±1.1	15.4±1.3	13.5±1.0	5.9±0.5	9.6±2.8
Hardware: ep (cm) ↓	15.9±1.4	17.1±1.7	17.8±1.5	8.6±0.6	13.1±3.9
Hardware: survival (s, cap 20)	20.0±0.0	20.0±0.0	20.0±0.0	20.0±0.0	19.0±2.0

A do-nothing floor, holding the nominal stance for the whole episode, reaches 28–30° of joint error on the three gaits, roughly double what CopyQuad leaves. The gestures are the clearest evidence that posture comes entirely from tracking the legs: base orientation is never commanded, yet the bow's stand-bow-stand cycle and the give-paw's three-legged balance are both recognizable on hardware. Every behavior is deployed to the physical robot zero-shot, with no fine-tuning, real-world data or gain retuning.

Matched keyframes for horse walk, dog walk, and horse trot: source video next to the simulated policy and the hardware rollout — Source video, simulated policy and hardware rollout at three matched phases, for the horse walk, dog walk and horse trot. Read left to right, each triplet reaches the same point in the gait cycle, with the trot's diagonal legs swinging and landing together while the walks move one leg at a time.

Matched keyframes for the dog bow and dog give-paw gestures: source video next to the simulated policy and the hardware rollout — The same comparison for the two non-periodic gestures. Both are recognizable from posture alone, on hardware as well as in simulation, even though nothing in the reward commands the robot's orientation.

Is the missing axis worth recovering?

CopyQuad pins the lateral axis instead of estimating it, so the natural question is whether a depth model would do better. Swapping the pin for three levels of depth, raw disparity, metric depth, and metric depth repaired to feasibility, answers it: raw disparity puts the robot on the ground within seconds, metric depth survives longer but still falls, and even repaired depth, the closest analogue to what per-clip optimization buys in prior work, never beats the constant pin.

Retargeting ablation bars for the dog walk and horse trot, comparing the pin against depth-based alternatives — Retargeting ablation on the dog walk and horse trot. Every variant is evaluated against the full method's own reference. Blue stays up and locomotes, red falls early, gray stays up but never travels. Three stages of depth machinery end up behind a single number read off the robot's own body.

The rigidity test: can a depth model even hold a constant distance?

The left-right hip separation a depth model implies is a rigid distance that cannot move, so a faithful model should return it as nearly constant, with no training or ground truth required to check. Neither of two recent, widely used models does. A relative video depth model (VDA) varies the implied separation by 40–107% of its own mean on the dog clips, and returns almost no separation at all on the horse clips. A single-image metric model (Depth Pro) roughly halves that swing, but still varies by tens of percent, and on the trot it puts the left and right hip on the wrong side of each other in two frames out of five, barely above what a coin flip would give. On the dog walk that is about 2 cm of frame-to-frame drift on a quantity that cannot move, set against the 4.5 cm peak-to-peak of true paw travel the motion itself contains.

What video costs against motion capture

Motion capture gives the one setting where the discarded axis can be checked against ground truth. Two routes run on the same underlying dog-walk motion: one retargets the 3D markers directly, the other renders that motion to video and recovers a reference with CopyQuad alone, discarding the 3D entirely. Both share the same retargeter, reward and training budget.

Route	eq (deg) ↓	ep (cm) ↓	Survival (s, cap 20)
MoCap route	9.5±1.0	10.5±1.5	20.0
Video route (CopyQuad)	12.5±1.4	13.5±1.8	20.0

The two routes differ by 3.0° and 3.0 cm, and both survive the full episode. That gap is the price of video at this tracking bandwidth, measured on a rendered clip with no blur, occlusion or compression, so it's a lower bound on the cost of real footage. The number that settles whether the gap matters is simpler: the true peak-to-peak lateral paw excursion of this walk is 4.5 cm, with a mean deviation near 1.4 cm, against the 3.5 cm of lateral foot error the policies already leave. Perfect lateral tracking would remove at most 1.4 of those 3.5 cm, under a tenth of the total foot error, so the discarded axis sits below the pipeline's own resolution.

Limitations

The design is scoped to sagittally dominated motion under a roughly static camera, conditions two in five casually chosen clips fail to meet. It's bounded to symmetric gaits: a canter's lateral excursion is comparable to the lateral foot error itself, which is well outside the regime the pin is built for. And the rigidity argument covers per-pixel monocular depth specifically, not per-clip articulated 3D reconstruction, which does hold the rigid axis, just at the cost of a per-clip optimization that this pipeline is built to avoid.

BibTeX

@article{danesh2026copyquad,
  title={CopyQuad: Quadrupedal Skill Transfer from a Single Side-View Video Without a Depth Model},
  author={Danesh, Mohamad H and Hu, Raina and Xu, Mengxue and Lin, Hsiu-Chin},
  journal={arXiv preprint arXiv:2610.16737},
  year={2026}
}

arXiv