Garbage In, Garbage Out: The Case For Better Robot Data Understanding

Low quality robot data → Poor robot performance.

Robot data collection is expensive, requiring hundreds of human expert teleoperation hours.

At the same time, collecting high quality robot data is difficult - even for a highly skilled teleoperator. For example, idle trajectories may occur when the teleoperator pauses or poor lighting might affect visual clarity.

While the precise definition of what constitutes a high quality training example is a complicated question (e.g. does a dark video increase policy resilience or reduce performance?), a few quality indicators can provide an insightful snapshot into your dataset. Data understanding is the first step to data improvement.

In this article, we introduce a lightweight Open Source toolkit to find low quality examples in the Open X-Embodiment datasets and show that training on 20% of low quality examples hurts policy training loss by 30%.

Want to know how good your dataset is?

Try out the tool here: https://github.com/RoboticsData/score_lerobot_episodes

---

Robot Data Defined

We ran experiments on the Open X-Embodiment datasets - a large collection of over 60 robot manipulation datasets gathered through teleoperation on robots such as Franka, UR5, and DROID.

A dataset is represented as a collection of N episodes (or trajectories): $$\mathcal{D}=\{{\mathcal{E}}_1,\dots ,{\mathcal{E}}_N\}\backslash mathcal\{D\}\; =\; \backslash \{\; \backslash mathcal\{E\}\_1,\; \backslash ldots,\; \backslash mathcal\{E\}\_N\; \backslash \}$$

Each episode consists of timestamped frames: $${\mathcal{E}}_i=\{(o_{i,t},s_{i,t},a_{i,t}){\}}_{t=1}^{T_i}\backslash mathcal\{E\}\_i\; =\; \backslash \{\; (o\_\{i,t\},\; s\_\{i,t\},\; a\_\{i,t\})\; \backslash \}\_\{t=1\}^\{T\_i\}$$

where $o_{i,t}o\_\{i,t\}$ is the observation, $s_{i,t}s\_\{i,t\}$ is the robot state, and $a_{i,t}a\_\{i,t\}$ is the action at time $tt$.

Observation space

Each observation $o_{i,t}o\_\{i,t\}$ includes two synchronized RGB camera streams:

1. An over-the-shoulder view showing the full scene and robot.
2. A wrist-mounted camera providing a close-up view of the robot’s end-effector.

State space

$$s_{i,t}=[q_{i,t},g_{i,t}]\in {\mathbb{R}}^{J+1}s\_\{i,t\}\; =\; [q\_\{i,t\},\; g\_\{i,t\}]\; \backslash in\; \backslash mathbb\{R\}^\{J+1\}$$ where $q_{i,t}\in {\mathbb{R}}^{J}q\_\{i,t\}\; \backslash in\; \backslash mathbb\{R\}^J$ represents the robot’s joint angles at time $tt$, and $g_{i,t}\in \mathbb{R}g\_\{i,t\}\; \backslash in\; \backslash mathbb\{R\}$ represents the gripper position (e.g., open or closed).

Action space

$$a_{i,t}=[q_{i,t}^{\mathrm{\prime }},g_{i,t}^{\mathrm{\prime }}]\in {\mathbb{R}}^{C+1}a\_\{i,t\}\; =\; [q\text{'}\_\{i,t\},\; g\text{'}\_\{i,t\}]\; \backslash in\; \backslash mathbb\{R\}^\{C+1\}$$ where $q_{i,t}^{\mathrm{\prime }}q\text{'}\_\{i,t\}$ are the motor commands and $g_{i,t}^{\mathrm{\prime }}g\text{'}\_\{i,t\}$ is the gripper command applied by the teleoperator.

Visual Scoring

We evaluate each episode for visual clarity indicators such as lighting and blur. This allows us to ensure that only high-quality visual data is retained for training.

Our process involves:

1. Uniformly sampling 10 frames per episode - This strategy provides a representative snapshot of the visual quality throughout the episode without the computational expense of analyzing every frame.
2. Compute per-episode aggregate visual score - For each frame $II$, we compute a penalty based on blur and brightness where frames which are too dark or too blurry are penalized. This penalty is then averaged across the episode and used to compute an aggregate visual score.
3. Remove episodes that score below a threshold - This final filtering step discards any episode whose aggregate visual score falls below a pre-defined quality threshold, ensuring only visually high-quality data is retained for training.

Blur

We estimate blur using the variance of the Laplacian. This metric is used because it measures the magnitude of high-frequency content in an image, which is drastically reduced by blur. The high-frequency components correspond to sharp edges and details; a sharper image will have a higher variance of the Laplacian.

The Laplace operator, $\mathrm{\Delta }\backslash Delta$, for a 2D image $I(x,y)I(x,\; y)$ is defined as the sum of its second partial derivatives: $$\mathrm{\Delta }I=\frac{{\mathrm{\partial }}^{2}I}{\mathrm{\partial }{x}^2}+\frac{{\mathrm{\partial }}^{2}I}{\mathrm{\partial }{y}^2}\backslash Delta\; I\; =\; \backslash frac\{\backslash partial^2\; I\}\{\backslash partial\; x^2\}\; +\; \backslash frac\{\backslash partial^2\; I\}\{\backslash partial\; y^2\}$$ In discrete form, it is often approximated using a convolution kernel e.g., a $3\times 33\; \backslash times\; 3$ kernel such as where $\mathrm{\Delta }I\backslash Delta\{I\}$ is the discrete convolution of the kernel $\mathrm{\Delta }\backslash Delta$ with the frame $II$.

The metric used for blur is the variance of the Laplacian $Var(\mathrm{\Delta }I)Var(\backslash Delta\; I)$, which is calculated across all pixels in the frame: $$Var(\mathrm{\Delta }I)=\frac{1}{HW}\sum _{x=1}^W\sum _{y=1}^H{(\mathrm{\Delta }I(x,y)-{\mu }_{\mathrm{\Delta }I})}^{2}Var(\backslash Delta\; I)\; =\; \backslash frac\{1\}\{HW\}\; \backslash sum\_\{x=1\}^W\; \backslash sum\_\{y=1\}^H\; \backslash left(\; \backslash Delta\; I(x,\; y)\; -\; \backslash mu\_\{\backslash Delta\; I\}\; \backslash right)^2$$ where $W\times HW\; \backslash times\; H$ is the frame size, and ${\mu }_{\mathrm{\Delta }I}\backslash mu\_\{\backslash Delta\; I\}$ is the mean of the Laplacian values over the frame $${\mu }_{\mathrm{\Delta }I}=\frac{1}{HW}\sum _{x=1}^W\sum _{y=1}^H\mathrm{\Delta }I(x,y)\mathrm{.}\backslash mu\_\{\backslash Delta\; I\}\; =\; \backslash frac\{1\}\{HW\}\; \backslash sum\_\{x=1\}^W\; \backslash sum\_\{y=1\}^H\; \backslash Delta\; I(x,\; y)\; .$$

We compute a blur penalty score by scaling the variance value, apply a threshold and return the proportion of blurred frames across the episode.

Here is an example of a blurry episode discovered by our tool

Example of motion and autofocus blur

Lighting

We use mean-brightness as a proxy for good lighting. If the mean intensity $\mu \backslash mu$ of the grayscale image is below 50 (on a scale of 0 to 255), we assign a linearly proportional penalty:

$$\text{Penalty}=\max (0.0,\frac{50.0-\mu }{50.0})\backslash text\{Penalty\}\; =\; \backslash max\backslash left(0.0,\; \backslash frac\{50.0\; -\; \backslash mu\}\{50.0\}\backslash right)$$

Note we only penalize images that are too dark; overexposure is not penalized since it is not a failure mode that occurs frequently and we observe that penalizing it can result in an excessive rate of false positives.

The following example illustrates the effects of our visual scoring function on the droid_100 dataset. The video that received a low score was one that was captured in a dark room.

Low lighting score due to dimly lit room

In many cases, the wrist camera suffers from a noticeable drop in visual quality compared to the overhead view due to occlusions, shadows and focus issues.

Motion Scoring

In addition to visual observations, each episode contains motion data representing the robot’s internal state and control commands over time. This includes joint positions, velocities, and actions, which specify the robot’s kinematic behavior.

We introduce scoring functions that classify the quality of motion of a given episode. We describe how we score for categories such as collision, path efficiency, and idle-time.

Collision

We aim to detect potential physical collisions by analyzing spikes in the robot's joint-space acceleration. The idea is that when a robot physically impacts an object, it experiences abrupt changes in joint motion—typically seen as sudden decelerations upon contact, and rapid accelerations as it moves away.

Our collision scoring function estimates an acceleration threshold for each joint, based on the median absolute acceleration over time. A spike is detected when any joint's acceleration at a given timestamp exceeds its respective threshold.

• Joint acceleration proxy:
  $$a_t=\frac{q_{t+1}-2q_t+q_{t-1}}{(t_{t+1}-t_t{)}^2}a\_t\; =\; \backslash frac\{q\_\{t+1\}\; -\; 2q\_t\; +\; q\_\{t-1\}\}\{(t\_\{t+1\}\; -\; t\_t)^2\}$$

• Per-joint robust threshold:
  $$\text{spike\_ratio}=\theta >15\times \mathrm{m}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{n}(\mathrm{\mid }a\mathrm{\mid })\backslash text\{spike\backslash \_ratio\}\; =\; \backslash theta\; >\; 15\; \backslash times\; \backslash mathrm\{median\}(|a|)$$

• Collision score (goodness):
  $$1-\text{spike\_ratio}1\; -\; \backslash text\{spike\backslash \_ratio\}$$

The function returns a score between 0 and 1 where a low-score indicates high likelihood of collision.

Path Efficiency

We also score path efficiency by calculating how close the motion of the robot is to a straight line. This is motivated by the fact that expert trajectories are often the most direct path between two points. A path with excessive meandering or corrections suggests poor human control or indecision, which introduces noise and complexity that hampers policy training and results in inefficient robot behavior.

Path length:
$$L=\sum _t\parallel q_{t+1}-q_t{\parallel }_{2}L\; =\; \backslash sum\_t\; \backslash lVert\; q\_\{t+1\}\; -\; q\_t\; \backslash rVert\_2$$

Straight-line distance:
$$D=\parallel q_{\text{end}}-q_{\text{start}}{\parallel }_{2}D\; =\; \backslash lVert\; q\_\{\backslash text\{end\}\}\; -\; q\_\{\backslash text\{start\}\}\; \backslash rVert\_2$$

Path Efficiency Score:

The final score is the ratio of the straight-line distance in joint space to the actual path length traveled.

There may be situations where the direct path between two points isn't the ideal trajectory. For example, manipulating an object or maneuvering past obstacles in the scene might result in high amounts of nonlinear local perturbations or the robot's joint configuration may not permit a direct path. In these cases, path efficiency may not be a useful metric and can be excluded from the scoring criteria.

Actuator Saturation

When the exptected action $a_{t}a\_t$ doesn't result in the next state $q_{t+1}q\_\{t+1\}$ we use this as a proxy for actuator saturation. Too much actuator saturation can indicate that the robot is lifting heavier loads than it is equipped for or is running into resistence within the scene or has faulty motors. In order to reduce robot wear and tear over time, we want to prevent this behavior inheriting this behavior from training data.

Specifically, we check how often $\mathrm{\mid }{a}_t-q_{t+1}\mathrm{\mid }>\text{threshold\_deg}|a\_t\; -\; q\_\{t+1\}|\; >\; \backslash text\{threshold\backslash \_deg\}$ for any joint (threshold default: 7).

To downweight minor transient divergence caused by slippage or inertia, we impose a non-linearity on the saturation ratio:

$$s_{\text{sat}}=\exp (-4\cdot \text{saturation\_ratio})s\_\{\backslash text\{sat\}\}\; =\; \backslash exp(-4\; \backslash cdot\; \backslash text\{saturation\backslash \_ratio\})$$

Idle Time

We score the robot’s motion based on how idle it is throughout the task. This is bad because periods of near-zero velocity are often the result of distraction, indecision, or taking a break rather than intentional action to complete the task. These stationary periods slow down learning and result in undesirable robot behavior.

Specifically, we calculate the proportion of time the robot's velocity lies below a certain threshold during its motion. We first calculate the joint-space velocity magnitude $\parallel {\mathbf{v}}_t\parallel \backslash left\backslash lVert\; \backslash mathbf\{v\}\_t\; \backslash right\backslash rVert$ for each time step $tt$: $$\parallel {\mathbf{v}}_t\parallel =\sqrt{\sum _{j=1}^J{\left(\frac{\mathrm{\Delta }{q}_j}{\mathrm{\Delta }t}\right)}^2}\backslash left\backslash lVert\; \backslash mathbf\{v\}\_t\; \backslash right\backslash rVert\; =\; \backslash sqrt\{\backslash sum\_\{j=1\}^\{J\}\; \backslash left(\; \backslash frac\{\backslash Delta\; q\_j\}\{\backslash Delta\; t\}\; \backslash right)^2\}$$

The raw score is $1.01.0$ minus the proportion of steps where $\parallel {\mathbf{v}}_t\parallel \backslash left\backslash lVert\; \backslash mathbf\{v\}\_t\; \backslash right\backslash rVert$ is below a threshold default of $0.10.1$.

We also note that filtering idle intervals improves performance in openpi

Idle time may sometimes be useful to attenuate inertia or allow the dynamics of the environment to settle to rest before the next action. In these cases, idle time is repeated and predictable rather than incidental.

We are currently in the process of launching a tool that accounts for these cases so if your use case exhibits such behavior, reach out to us!

---

Corruption Experiment

We evaluate the effect of noisy examples on robot learning by artificially corrupting the robot using the sources of noise we aim to identify using our tool. Specifically, for 20% of episodes, we introduce the following realistic corruptions:

• Visual Noise: We artificially darken frames and apply an unsharp mask or motion blur to a random subset of frames in the episode.
• Motion Noise: We introduce short, random idle periods (setting joint velocities to zero for 0.5-1.0 seconds) and insert single-step acceleration spikes (random, large joint command outliers) to simulate teleoperator jerks or minor collisions.

The following is the side by side comparison between an original video of an episode and the corrupted version.

Note on evaluations

In a typical machine learning experiment, the most direct approach would be to compare the held-out validation loss on a model train on the filtered and unfiltered versions of a real-world dataset.

However, in the context of robotics, validation loss isn't representative of real-world performance and the training loss is not directly comparable since the filtered set has fewer examples.

For simplicity's sake, we opt to motivate the importance of data understanding by using an artificial corruption experiment to illustrate the impact of low quality data on robot learning.

We invite the community to share real world or simulation evaluation results and plan to share more of our own results in a future post.

Retrieval Precision-Recall

Just for illustrative purposes, we can design the following recall/percision experiment where we use our scoring algorithm to detect corrupted episodes within a dataset that has been corrupted. Essentially, we have the following binary classification problem where we set a threshold where episodes whose score fell below this result where classifed as corrupt.

For example, we have the following results for the detecting corrupted videos using the visual scoring function in the following Stanford HYDRA dataset. This is a robotic manipulation dataset containing 570 episodes of Franka robot demonstrations across 3 tasks, with synchronized wrist and external camera views and 7-DOF action sequences.

For this experiment, we corrupted 50% of these episodes and obtained the following precision and recall results for various thresholds.

Precision, Recall and F-1 curves on the Stanford HYDRA dataset (50% corrupted).

Effects on Training

We also compared the training losses between the original datasets and the corrupted datasets where we used ACT as our policy.

The figure below compares policy training on a corrupted dataset—where 20% of the episodes were intentionally degraded—with training on the original, clean dataset.

We see that models trained on corrupted data require significantly more optimization steps to reach the same loss threshold as those trained on clean data, highlighting the sensitivity of policy learning to data quality.

Standard Hydra training loss curve.

Berkeley Autolab UR5 training loss curve.

In conclusion, these results quantify the large impact of incorporating low quality data into an otherwise useful dataset, resulting in poor performance and wasted GPU hours.

---

We'd love to hear from you - do send us any comments or feedback you have!

https://github.com/RoboticsData/score_lerobot_episodes