What Data Do Robot Foundation Models Need?

A robot foundation model is only as general as the data behind it. Many current approaches share a broad pattern: a vision-language backbone paired with an action model, trained across tasks and robot platforms, then adapted to specific tasks. The data mix is becoming an increasingly important differentiator. This post explains what that mix contains, how much public data exists, and what a dataset needs before it is useful for training.

What is a robot foundation model?

A robot foundation model is a generalist policy trained on demonstrations from many tasks, scenes and often many robot bodies, so it can be adapted to new tasks with relatively little extra data. Most current examples are vision-language-action (VLA) models: they take camera images and a text instruction and output robot actions. Two public examples show the pattern. The Open X-Embodiment collaboration pooled data from 22 different robots across 21 institutions, covering 527 skills, and showed that a model trained on the pooled data improved the capabilities of several robots by drawing on experience from other platforms. For Physical Intelligence's π0, the authors report pre-training on over 10,000 hours of robot data, with results covering tasks such as laundry folding, table cleaning and assembling boxes.

Three kinds of data in the mix

Most training mixes combine three sources of physical data, alongside the web-scale image and text data inherited from the vision-language backbone.

The Universal Manipulation Interface paper describes the middle option as a framework for transferring skills from in-the-wild human demonstrations directly to robot policies. Egocentric video can be collected without deploying a robot, which may make it easier to scale in some settings; teleoperation is the most direct. Many programmes combine more than one.

How much public data exists

The public datasets cover different kinds of data and tasks, so they aren't directly comparable. Even the large ones remain modest relative to the scale and diversity sought by today's robot-learning programmes.

• DROID contains 76,000 demonstration trajectories, or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia and Europe over 12 months.
• Ego4D contains 3,670 hours of daily-life video from 931 camera wearers in 74 locations across 9 countries.
• The π0 authors report that 9.1% of their training mixture consisted of open-source datasets, including Open X-Embodiment, Bridge v2 and DROID.

The last figure is the telling one: a team with access to the main public datasets still built most of its training mix from its own data. For a company building a foundation model, data collection is part of the job, not a side task.

Variation matters as much as hours

Hours are the easy number to quote, but a model generalises from variation: different rooms, lighting, objects, tools, operators and ways of doing the same task. DROID's authors made scene diversity the point of their dataset, collecting in 564 scenes rather than a few lab benches. The same logic applies to where the work happens. Look at where the large public datasets were recorded: research labs, offices, homes and daily life. Industrial and commercial work done at working pace (sewing, packing, assembly, kitchen prep, laundry) is thin by comparison. A model that has never seen a cluttered workbench or a production-line rhythm meets them for the first time in deployment. We have written about this as the distribution shift problem.

Where human video fits

Human video is attractive because people already do the work, everywhere, and a camera is cheap compared with a robot. The open question has been whether it helps a robot policy at all. The EgoMimic results suggest it can. In their experiments, which pair egocentric video with 3D hand tracking and robot data, an additional hour of human hand data was more valuable than an additional hour of robot data. That is a finding about tracked hand data in their setup, not about unannotated video in general. That result comes with conditions. Egocentric video carries no robot actions, so it needs a bridge: hand and object tracking, a shared representation with robot data, or a smaller set of robot demonstrations of the same task. Treat it as a way to add breadth cheaply, with teleoperation or UMI data anchoring the robot's own actions. Our guide to egocentric data collection covers capture setups and what a delivery should contain.

What a usable dataset needs

Raw video is not a training set. Before data is useful for pre-training or fine-tuning, check that it has:

• Episode boundaries that mark where each attempt at the task starts and ends.
• Task descriptions in plain language for each episode, since VLA models are conditioned on instructions.
• Synchronised timestamps across cameras and any other sensors.
• Camera calibration and mounting details, to document the sensor geometry and viewpoint.
• Variation metadata: scene, objects, operator and conditions per episode, so you can measure coverage instead of guessing it.
• Success and failure labels, or at least a review status, for each episode.
• Actions or proxies where they exist: robot actions for teleoperation, gripper pose and width for UMI, hand tracks for human video.
• Provenance: who recorded it, where, with what permission and consent.
• Licence terms that cover the files, the annotations and your trained models. See our post on exclusive and non-exclusive dataset licensing.

Questions to ask before you collect or buy

1. Which tasks must the model learn, and which can it borrow from public data?
2. What variation does the model need: how many scenes, objects and operators per task?
3. Which capture method fits each task: teleoperation, UMI or egocentric video?
4. How will episodes be reviewed, and what counts as accepted?
5. Can the provider document where and how the data was collected?
6. What does the licence let you do with the data and with the models trained on it?
7. What does a pilot cost, and what will it prove? Our guide to robot training data costs lists the factors that drive price.

Where Telepath fits

Telepath Robotics collects robot training data in real workplaces in the United Arab Emirates and Central Asia: egocentric video, UMI demonstrations and teleoperation data, scoped to the task your model needs to learn. Every project starts with a pilot, so you can check the data against your training pipeline before committing to volume.