This article is about reinforcement learning (RL). DAgger in the previous post taught the model from the expert rescue skill's corrections. Now v4 succeeds on its own some of the time, so every run gets a grade, a simple reward, and the next model learns most from its own runs that went best.
What does this article cover?
How runs are graded, how 50 graded v4 runs became the v5 dataset, and why v5 did worse.
| Version | Training data | Loss | Success rate | Performance | Root cause | So the next dataset |
|---|---|---|---|---|---|---|
| v4 | v3 data + 25 DAgger corrections (previous post) | 0.016 | 8 / 50 (16%) | Picked up the tube on its own, the first model to do so | Learned only from demos and corrections. Its own good runs were never used | grade every run, add v4's successes weighted by grade |
| v5 | v4 data + v4's own successes (weighted) + round 2 corrections | 0.022 | 5 / 50 (10%) | Worse than v4. Hovered over the tube and rarely closed the gripper (11 of 50 runs) | The grade copied v4's slow, hovering successes 2 to 3x, and the old expert rescue skill's corrections were stop-and-look moves. Both taught hovering | keep v4's data, add the next round's corrections from the improved expert rescue skill, and score runs with a hover penalty |
Success rate counts the runs where the model lifted the tube with no help.
What does the grader add on top of DAgger?
The grader is the watcher from the previous post with one more job: at the end of every run it writes down how the run went. Training then uses that grade as a weight. Runs that went well count more, runs that failed count little or nothing. This is the simplest form of learning from reward, the idea behind reinforcement learning, done offline on recorded runs instead of live trial and error.
It needs the model to succeed sometimes. With v3 at 0 of 30 (0%) every grade would have been a zero and there would be nothing to weight up. v4 is the first model where grading pays off.
How is RL with grades different from DAgger?
Both improve the model from its own runs, but they learn from different things.
| DAgger (previous post) | RL with grades (this article) | |
|---|---|---|
| Learns from | Someone else's moves: the expert rescue skill's corrections | The model's own moves, from its runs that went well |
| Signal | "Do this instead", a demonstration | "That run was good", a score called a reward |
| When it helps | The model fails and needs to see how to finish | The model already succeeds sometimes and needs to do the good version more |
| Weak spot | Only as good as the expert | Only as good as the reward: a careless grade rewards the wrong habit |
| Form used here | Keep the expert's frames | Offline RL called reward-weighted or filtered behavior cloning: copy each good run more times the higher its grade |
The two are combined: DAgger keeps supplying corrections for the runs that fail, and the grade decides how much each successful run counts.
The DAgger loop from the previous post, plus grades. Same watcher and expert rescue skill. The new part is the grade on every run, which decides how much each successful run counts in training.
How is each run graded?
The watcher checks the arm 30 times a second while the model drives and ends the run with one of these outcomes.
| Outcome | Rule | Grade |
|---|---|---|
| Success | Gripper partly closed (between 3 and 22) and the tip more than 3.5 cm above the table for 20 frames in a row. The watcher then raises the shoulder until the tip is about 9 cm up, longer than the tube, so a tube held at one end hangs free. Last, it squeezes slightly: the jaws must stop on the tube. | Pass |
| Empty grip | Lifted as above, but the squeeze closes on nothing. | Fail |
| Stall | After the first 4 seconds, the gripper tip moves less than 1.2 cm and the base turns less than 3° over 10 seconds. From episode 39 on, 4 seconds, to save time. | Fail |
| Timeout | 45 seconds without a success. | Fail |
Each run's record also keeps who finished it (the model alone, the expert rescue skill, or nobody), how long the model drove, and how high the model lifted the tube by itself. The servos do report load, but the tube weighs about 15 grams, likely too little to show against the arm's own weight. That hasn't been measured yet, so the grade does not use it.
How was the v5 dataset built?
v4 data + v4's own successes (weighted by grade) + round 2 corrections.
DAgger round 2: grading 50 runs of the v4 model
v4 from the previous post drove 50 runs with the watcher on, and every run was graded. It lifted the tube on its own in 23 runs, and those graded passes went into the v5 data. This round let the arm pause up to 10 seconds before calling a stall, longer than the 4 seconds used to compare models, so some slow runs finished on their own here.
A graded pass, about 1.7× speed (5 s). v4 lifts the tube on its own in 8 seconds. Runs like this are the new ingredient in v5.
| Outcome | Ep. 0–32 | Ep. 33–38 | Ep. 39–49 | Total |
|---|---|---|---|---|
| Pass: v4 lifted the tube on its own | 19 | 0 | 4 | 23 |
| Fail, the expert rescue skill rescued it | 9 | 2 | 7 | 18 |
| Fail, rescue failed too | 5 | 4 | 0 | 9 |
v4's passes took 8 to 32 seconds, with a median of 13. Episodes 33 to 38 stand apart: a failed rescue had left the tube close to the base, every grip there missed, and neither v4 nor the expert rescue skill could get it out. From episode 39 the stall window was shortened to 4 seconds, which hands over sooner and so counts some slow successes as rescues.
What goes into v5
v4 learned from human and scripted demos plus the expert rescue skill's corrections. It never saw a model succeed on its own, because v3 never did. v5 is the first round where the model's own successes go in, and where the grade decides how much each run counts.
| From the v4 round | Into v5? | Why |
|---|---|---|
| Pass | Yes, 2 copies, or 3 for runs faster than the median | Good behavior in the model's own style. Repeating it is the grade weighting, done without changing the trainer. |
| Fail, the expert rescue skill rescued it | Only the expert rescue skill's correction frames | Shows how to finish from where the model got stuck. v4's frames before the stall get weight 0. |
| Fail, rescue failed too | No | The expert rescue skill's failed moves are not demonstrations. They go to fixing the watcher and the skill. |
| Episodes 33 to 38 | No | The near-base streak. The starts were unusual and the runs mostly failed. |
| Everything v4 trained on | Yes, unchanged | The demos and the round 1 corrections still hold. |
Idle stretches are trimmed to 3 frames as before. Each episode is checked for missing or non-finite values, and for matching frame counts across both cameras and the joint data.
The v5 dataset
| Episode type | Source | Runs | Episodes in training |
|---|---|---|---|
| Human teleop demo | Series 3 (v1) | 20 | 20 |
| Scripted demo | Series 3 (v2) | 27 | 27 |
| Fail + rescue, old skill | DAgger round 1 (v3) | 25 | 25 |
| Fail + rescue, old skill | DAgger round 2 (v4) | 16 | 16 |
| Model success, graded | DAgger round 2 (v4) | 23 | 58 |
| Total | 111 | 146 |
Runs are distinct recordings. Episodes in training counts the extra copies of graded successes. In frames: 62,095 in total, made of v4's 72 episodes (33,180 frames), 23 passes as 58 weighted copies (9,670 frames, 23,143 with copies) and 16 rescues' correction frames (5,772). Left out: 6 near-base episodes, 5 failed rescues, and v4's own frames before every stall.
How was model version 5 trained?
On the v5 dataset above, 20,000 steps from the SmolVLA base model with the same settings as v4, on a Runpod A40 at $0.59/h. That is about 1.8 s per step, about 10 hours and $6. Final loss 0.022. The 10,000-step checkpoint was also tested, with the same result (12%), so more training wasn't the issue.
How did v5 do over 50 runs?
Same 50-run test as v4. v5 alone: 5 of 50 (10%).
v5 hovering over the tube, about 3× speed (5 s). It reaches the tube and lines up, then hovers above it for 16 seconds without closing until the watcher calls it stuck. This is the habit the coarse grade rewarded.
Results
| Measure | v4 | v5 |
|---|---|---|
| Model picked up the tube alone | 8 / 50 (16%) | 5 / 50 (10%) |
| Runs where the model closed the gripper at all | 30 / 50 (60%) | 11 / 50 (22%) |
| Hover before closing (median) | 6.8 s | 8.7 s |
| Expert rescue skill rescued the rest | 18 / 27 (67%) | 43 / 45 (96%) |
The other v4 figures come from the graded round above. Hover is the seconds the jaws spend open within 3 cm of the table before they start closing. The rescue rate rose because the expert rescue skill was improved during these tests (see the appendix of the previous post), not because of the model.
What it says about RL
v5 was the first test of reinforcement learning (RL) in this series, in its simplest offline form: grade the model's own runs and train on the good ones, weighted by grade. In robot learning this is called reward-weighted or filtered behavior cloning. The result is a clear no for this version: the model got worse.
That doesn't settle whether the method works, because the reward was too coarse. It only said "lifted the tube, and was it faster than the median". v4's successes hovered for about 7 s before closing, so every success copied 2 or 3 times also taught the model to hover, and the grade never counted that against it. The model learned the habit and dropped the commitment: it now hovers and often never closes.
Root cause
- A reward that ignored hovering. Slow, hesitant successes got the same 2 to 3 copies as clean ones.
- Corrections that hovered too. The round 2 corrections came from the old expert rescue skill, which lined up in stop-and-look steps right above the tube. Trimming removed the pauses but kept the small moves between them.
- Two changes at once. v5 changed the data in both ways together, so the test can't separate them. Both point the same way.
What comes next?
The v5 test wasn't wasted. Its 50 runs were also the third round of DAgger, and this time the corrections came from the improved expert rescue skill, which took over from where the model stopped in one continuous motion with no hovering. It rescued 43 of the 45 failures.
That sets up the next model, v6. It keeps the RL idea and fixes both inputs that went wrong: the hovering corrections are replaced by the continuous ones, and the grade becomes a score that counts hovering against a run instead of only asking whether the tube was lifted. The next post covers how that data was built and how v6 did.