SAKI Paper

SAKISkill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation

Yijie Lu1,† James Zhao1,† Weiming Zhi1,2,*

1School of Computer Science and 2Australian Centre for Robotics, The University of Sydney, Australia

†Equal contribution. *Corresponding author: Weiming.Zhi@sydney.edu.au

Watch the film Paper arXiv BibTeX
Input
One human video per skill
Kept from the video
Object motion, grasp, persistent contact, how the action ends
Output
Coordinated base, arm and gripper motion on a mobile robot
Shown here
Reuse in new layouts, assembled long-horizon tasks, contact-rich tasks
01Learn↓

One human video per skill

Learn the interaction

SAKI recovers object motion from a single, imperfect demonstration and keeps what the task needs: how the object is held, where contact persists and how the action ends. The robot’s own motion is solved later, in its own scene.

Human1×
Point-cloud reconstruction of the desk and the bin
Recovered can path
Dispose of a can. The line is the can’s observed centre, drawn as the demonstration plays.
Human1×
Point-cloud reconstruction of the whiteboard and the table
Recovered eraser path
Wipe the whiteboard. The dashed segment bridges a gap with no observations while the eraser is carried.
Human1×
Point-cloud reconstruction of the table with the tray, tape and basket
Recovered cup path
Pour from a paper cup. The line is the cup’s observed centre from grasp to release, drawn as the demonstration plays.
Skill preparation: recover observations (track, reconstruct, motion), interpret the interaction (roles, phases, contact requirements), and prepare reference and affordance into an object-task asset.
From video to a reusable skill: roles, phases and contact requirements guide reference and grasp preparation.
02Reuse↓

Works in layouts it has never seen

AutonomousNine recorded trials

Reuse across scenes

A frozen skill is bound to the objects in front of the robot. Whole-body kinematic imitation plans base, arm and gripper motion for the new layout, and accepted visual updates revise what remains without undoing committed motion.

Six recorded pouring layouts, their initial cup and tray positions, and recorded tool paths at a common metric scale.
One frozen pouring skill across changed cup, tray and robot start positions. Six of twenty trials shown.
Plots of a 7.12 cm shift in the bin estimate, the resulting target correction over task progress, and previous versus updated base plans.
A 7.12 cm shift in the bin estimate revises the remaining targets and the accepted base plan.
03Assemble↓

Separate skills, one longer task

Autonomous8×

Tidy and wipe

Skills from independent demonstrations are selected, ordered and bound to objects in a shared workspace. Scene estimates carry across skill boundaries, and each stage starts from the robot configuration the previous one left.

Autonomous16×
Collect a ball, carry the cup and pour
Autonomous12×
Carry a basket, collect at two tables
04Contact-rich↓

Contact held through the whole interaction

Demonstration → robot

Some interactions are defined by what happens after the grasp. SAKI keeps the eraser pressed and aligned to the board, couples a door handle to the door’s motion, and holds a lid open with one arm while the other places an object.

Human1×
Autonomous4×

Wipe the whiteboard. The eraser stays pressed and aligned to the board as it moves.

Demonstration and robot run are separate recordings, shown side by side, not synchronised.

05Method↓

How SAKI works

SAKI overview. Left: object-task assets are selected, ordered and bound to the current scene, giving scene-grounded object motion for three numbered stages. Right: whole-body optimisation turns planning inputs and scene-grounded targets into coordinated execution, with a closed loop through updated observations, persistent object state and updated remaining targets.

  1. Assemble

    Select, order and bind object-task assets to the goal and the current scene.

  2. Imitate

    Optimise whole-body motion so the object follows its scene-grounded targets.

  3. Correct

    Keep object state across viewpoints and revise remaining targets from new observations.

06Film↓

The full film

Every task above, in context. 2 min 59 s, narrated, with captions.

Read the transcript

0:05 The problem. Each skill uses just one demonstration video, despite noise, viewpoint changes and occlusions. The robot works in a different scene, with its own body and new object locations. How can it preserve the interaction and connect these demonstrations into a complete task?

0:21 Learn the interaction. We recover object motion from human video and prepare reusable skills. Each skill retains the interaction that matters: how an object is held, where contact is maintained, and how the action ends.

0:33 Reuse across scenes. The interaction stays reusable while robot motion adapts to the current scene. Object roles locate the targets, and coordinated base and arm motion realises the task. New observations can revise the remaining motion.

0:50 Assemble a task. These skills can be selected, ordered, and bound to objects in a shared workspace. Can disposal, tape placement, and wiping then become stages of one connected mobile task.

1:05 Tidying and wiping. Separate demonstrations can form one continuous task. Skill boundaries retain scene context as the robot moves between objects. Each stage continues from the preceding robot configuration. Different interaction requirements are handled within the same sequence. Retained scene estimates provide the context for the next action.

1:33 Collection and pouring. Long-horizon tasks connect object interactions across locations. Loading establishes the conditions for transport and pouring, linking several skills into one continuous execution.

1:45 Multi-location collection. One arm keeps hold of the basket while the other collects objects. The robot carries the growing collection between tables, reusing the same interaction at successive locations.

2:03 Box opening and placement. Some tasks require overlapping roles. Bimanual coordination lets one arm maintain support while the other manipulates an object. Contact roles overlap, and support persists until the other arm has safely withdrawn.

2:22 Whiteboard wiping. Wiping requires contact to persist as the tool moves across the surface. The transferred skill preserves tool alignment while the robot adapts its motion to the scene.

2:43 Door opening. Door opening couples engagement with the handle to the motion of the door, extending the interaction beyond the initial grasp.

2:52 Closing. SAKI connects reusable human interactions with scene-dependent robot motion, enabling longer mobile manipulation tasks.

07Abstract↓

Abstract

Learning from human videos offers a promising route to acquiring diverse manipulation skills. Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations. We present Skill Assembly and Kinematic Imitation (SAKI), a framework connecting human-video skill acquisition, cross-demonstration assembly and closed-loop whole-body execution.

SAKI prepares reusable object-centric skills that preserve task-critical interactions while allowing transfer paths to adapt. Given a goal and supplied task dependencies, it selects and orders skills, binds their object roles to the current scene, and carries scene estimates and robot configuration between successive skills. Whole-body kinematic imitation generates coordinated base, arm and gripper motion. During execution, persistent object estimates maintain task references across viewpoint changes, while visual feedback updates remaining trajectories.

Real-robot experiments demonstrate skill reuse across layouts and the composition of independently demonstrated interactions into continuous mobile tasks, including tidying and wiping. Ablation results show that task-conditioned reference preparation substantially improves long-horizon task completion with whole-body optimisation and visual feedback held fixed.

08Citation↑

Cite SAKI

Download .bib
@article{lu2026saki,
  title         = {{SAKI}: Skill Assembly and Kinematic Imitation from Human Videos
                   for Long-Horizon Mobile Manipulation},
  author        = {Lu, Yijie and Zhao, James and Zhi, Weiming},
  journal       = {arXiv preprint arXiv:2609.36031},
  year          = {2026},
  eprint        = {2609.36031},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}

Reusable interactions.
Connected tasks.

Read the paper Watch the film

SAKI film

Download 1080p

Choose a chapter, or play from the beginning.