Egocentric Data Collection: A Practical Guide for Robotics Teams

Egocentric data collection means recording first-person video from a camera worn by someone doing a real task, then organising that footage so a robot learning system can train on it. The camera sees roughly what the wearer sees, which in practice means their hands, whatever they're holding, and the order they do things in. That's the appeal for robotics teams. You can show a model how physical work actually gets done without putting a robot anywhere near the job. Below we go through what the data is, where it helps and where it doesn't, how a collection gets planned, and what to check before paying for one.

What egocentric data is

The camera sits on the head, in a pair of glasses, or on the chest of the person doing the work. Third-person video is the opposite arrangement: a camera across the room, filming the person from outside. For manipulation, the difference is bigger than it sounds. A first-person camera keeps the hands and the object in shot at the moment of contact, from an angle not far off where a robot's head or wrist camera would be. A camera in the corner of the room often gets the back of someone's shoulder instead. Video is the core of it, but depending on the device a recording can carry more:

• IMU data showing how the head or body moved
• audio, which is often stripped out or restricted for privacy reasons
• eye gaze, if the device tracks it
• depth or stereo video from multi-camera rigs
• labels added afterwards, such as task steps, hand and object boxes, or spoken narration

You don't need all of it. Whether an extra stream is worth paying for comes down to what your model will actually use.

Why robotics teams use egocentric video

Robot demonstrations are slow and expensive to gather. Every one needs a robot, an operator and time on a working setup. Human video needs none of those things, so a single collection can take in far more people, objects and places, and far more ways of doing the same job. There's good research on how that variety carries over to robots. R3M pre-trained a visual representation on Ego4D, a large human video dataset. Across 12 simulated manipulation tasks it improved task success by more than 20% over training from scratch and by more than 10% over representations like CLIP and MoCo, and it let a real Franka arm learn tasks in a cluttered apartment from 20 demonstrations. EgoMimic went a step further and trained a single policy on human and robot data together, with the human side recorded on Project Aria glasses. In its experiments, an extra hour of human hand data was significantly more valuable than an extra hour of robot data. There's a plainer benefit too. When a model sees one task done by lots of different people in lots of different rooms, it starts to pick up which parts of the job never change and which parts do. The catch is that egocentric video has no robot actions in it. No joint positions, no gripper commands, no force readings, because there was no robot. That's why most teams pair it with demonstrations that do include actions.

Egocentric, UMI and teleoperation data compared

Egocentric video is one of three common ways to capture robot training data. Each one answers a different question, and plenty of projects use two of them on the same task.

The Universal Manipulation Interface, or UMI, sits somewhere in between. Someone performs the task with a handheld gripper that has a camera mounted on it. The result looks a lot like what a robot gripper would record, and nobody had to bring a robot to the site.

Public egocentric datasets, and where they stop

Most teams start with one of the big public datasets, and with good reason.

They're excellent for pre-training and research. Where they tend to run out is when you're building something specific:

1. Task coverage. Your task, done the way your customer does it, probably isn't in there, or shows up only a handful of times.
2. Environment. The place your robot will work, with its particular objects, layouts, lighting and mess, usually isn't represented either.
3. Licence. Non-commercial terms or limits on redistribution may not fit a commercial model. Read the agreement you'd actually be signing, not someone's summary of it.

That's usually the point where a team starts looking at commissioning its own data.

How an egocentric data collection runs

You plan a collection backwards, starting from what the model has to learn. It tends to break into four stages:

1. Define. Write down the task, the variations you care about (objects, layouts, people, lighting, sites), how much data you want, the capture setup, and what counts as acceptable. Agree what a usable episode looks like before anyone presses record.
2. Collect. Line up the sites and participants, set up the cameras, and record the variations you planned.
3. Review. Check each recording against the acceptance criteria. Flag the ones that miss the brief, look for gaps in coverage, and go back for another pass where needed.
4. Deliver. Hand the data over in the structure and format you agreed, with the metadata the brief asked for.

Run a small pilot first. It puts the camera setup, the brief and the review criteria in front of real recordings while they're still cheap to change.

Decisions to make before anyone records

A lot of what goes wrong with egocentric data is settled before the first recording, either avoided or built in.

Camera placement

Head-mounted cameras and camera glasses follow where the wearer is looking, so for bench and table work they usually keep the hands in shot. Chest mounts give steadier footage but tend to lose the hands when someone reaches up or works close to their body. Pick based on where the work happens and, if you can, on where your robot's camera sits.

Hands in frame

For manipulation, the footage is only useful if the hands and the object are visible at the moment of contact. Field of view, mounting angle and what you tell participants all play into this. Don't assume it; measure it in the pilot.

Resolution and frame rate

Higher frame rates handle fast hand movement better, and higher resolution helps with small parts and fine work. Both cost you in storage and processing, so set them from what the model needs rather than from the best the camera can do.

Variation by design

If one person records everything in one room, the model learns that person and that room. Plan the variation up front, tag each episode with the variation it covers, and check what came back against the plan.

Episode boundaries

Decide where an episode starts and stops, and whether failed or interrupted attempts are kept and labelled. Often the recovery from a mistake is the most useful part of the clip.

What you should get at the end

Formats vary, but a usable delivery lets your team find things, filter them and trust them. At a minimum you'd want:

• video split into episodes that line up with the task definition
• metadata for each episode: the task, the site type, a participant ID and the variation tags from the brief
• any extra streams the brief asked for, synced to the video
• a manifest listing every file and episode
• a coverage report showing how the episodes map to the planned variations, and what was flagged or re-recorded

Agree the folder structure and file formats in writing before collection starts. Settling them late usually means reformatting everything.

Questions to ask any vendor about consent, privacy and provenance

Egocentric video records real people at real workplaces, so it's fair to ask where the footage came from and how it was gathered. Before signing anything, we'd ask:

• Consent. How did the people wearing the camera agree to be recorded, and can they withdraw?
• Site permission. Did the owner or operator of each site sign off on the recording?
• Bystanders. What happens with other people who wander into shot?
• Faces and personal details. Are faces blurred? Are screens, documents and ID cards kept out of frame or removed afterwards?
• Audio. Is it recorded at all, and if it is, how are conversations dealt with?
• Origin. Did the vendor collect the data itself or buy it from someone else, and is that labelled on each dataset?
• Local law. Which data protection law applies where the footage was shot? In the UAE, for instance, the main federal law is Federal Decree-Law No. 45 of 2021 on the Protection of Personal Data, in force since 2 January 2022. Free zones such as the DIFC and ADGM run their own regimes.

Get the answers in writing.

Where collections go wrong

Too little variation is probably the most common problem: plenty of hours, but from a handful of people at one site, so the dataset looks big and teaches a narrow model. Hands drifting out of frame is another. It's easy to catch in a pilot and frustrating to discover afterwards. Then there are problems of definition. If nobody decided where episodes start and end, the data is hard to segment, label and evaluate. If nobody defined "usable" in advance, review turns into an argument. Sometimes the problem is expectation. Human video on its own rarely gets a specific robot to a policy you can deploy, so plan to pair it with UMI or teleoperation data that includes actions.

Questions robotics teams ask

How much egocentric data does a robot learning project need?

It depends on the model and the task. The public research datasets run to thousands of hours, but if you're fine-tuning on one task you rarely need to start anywhere near that. Run a pilot big enough to test the setup and the brief, then scale up whichever variations turn out to make a difference. Settle the full volume after the pilot, not before.

Can egocentric data train a robot on its own?

Not usually, if you want a policy you can deploy. It shows a model what a task looks like and how hands and objects move, but there are no robot actions in it. Most teams use it for pre-training or co-training, alongside UMI or teleoperation demonstrations from their own setup.

Do I need special hardware?

No single device is required. People use head-mounted action cameras, camera glasses and multi-camera rigs. The choice comes down to field of view, which extra sensors you need, battery life for long tasks, and whether the wearer can put up with it for a full shift.

What format should the data be delivered in?

Whatever your training pipeline already reads. Pin down the episode structure, file formats, metadata fields and naming in writing before anyone starts collecting.

Getting started

If a public dataset already covers your task, use it. If it doesn't, the next step is a short written brief: the task, the variations you care about, what counts as acceptable, and the format you need. That's enough to scope a pilot. Telepath runs egocentric data collection for robotics teams in real workplaces, scoped from exactly that sort of brief. If you're still deciding between egocentric video, UMI demonstrations and teleoperation data, the robotics data collection overview compares all three.