The same tube and the same 20 demos as before, but this time the model starts with knowledge from thousands of other people's robot demos. Does a head start fix what broke ACT in the second post?
What does this article cover?
Three versions of SmolVLA fine tuned on demos only, what each one could do, and what its failure taught me about the next dataset.
| Version | Training data | Loss | Success rate | Performance | Root cause | So the next dataset |
|---|---|---|---|---|---|---|
| v1 | 20 human demos | 0.009 | 0 / 3 (0%) | Found the tube and reached for it, but grabbed beside it | Too few examples of the final grip, from too few tube positions | added 27 scripted demos from many more positions |
| v2 | 47 demos (20 human, 27 scripted) | 0.016 | 0 / 2 (0%) | Lined up right over the tube, but froze there and never tried to grab | The skill paused too long before each grip, so the pauses swamped the short gripping motion and the model learned to hover instead of grab | trimmed every pause to 3 frames |
| v3 | Same 47, pauses trimmed | 0.014 | 0 / 30 (0%) | Closed the gripper for the first time (v2 never did), but the jaws shut beside the tube | Demos show the ideal path, never how to recover from where the model actually ends up | let the model drive and recorded an expert correcting it (DAgger) |
Success rate counts the runs where the model lifted the tube with no help.
What is SmolVLA?
| Made by | Hugging Face's LeRobot team, released June 2025 (model, paper) |
| Size | About 450 million parameters, roughly 9 times ACT and small enough to run on a Mac |
| Built on | SmolVLM, Hugging Face's small vision language model, with an action head added that outputs arm motion |
| Pretrained on | About 480 community datasets on the Hugging Face Hub, roughly 23,000 episodes, mostly arms like mine doing table tasks |
| Input | Camera images, joint angles and a text instruction: "Pick up the tube by the cap and lift it." |
ACT started from zero and learned only from my demos. SmolVLA has already seen many arms like mine pick and place things. The question is whether that experience helps it handle a tube in a spot my demos never covered.
What setup and tools did I use?
It is the same table as the first two posts: the follower arm, a wrist camera, an overhead camera on a boom, and a desk lamp. The 20 demos were recorded with the leader arm for the ACT post and are reused here.
| Tool | What it does here |
|---|---|
| LeRobot | Hugging Face's open source robotics toolkit. It records the demos, trains the model with one command, and runs the trained model on the arm. |
| Hugging Face Hub | Hosts the SmolVLA base model, my datasets and each fine tuned model, all private, so the rented GPU can download them. |
| Runpod | Rents GPUs by the second. Claude connected to it through Runpod's Claude Code plugin, so it can check prices and stock, start and stop machines, and read the bill on its own. I used an H100 for v1, an L40 for v2 and an L40S for v3. |
| Claude Code | Claude Opus 5.5 on my Mac mini runs everything: the arm, the scripted skill, the uploads, the GPU machines, and fixing them when they break. I steer it from Telegram, mostly while away from home. |
| Mac mini (M4) | Drives the arm and the cameras, and runs the model during tests at about 30 commands a second. |
How is the training data prepared?
Every dataset goes through the same steps before it reaches the GPU. The checks are all rules on the recorded joints and run results, and nobody labels frames by hand.
- Record. Each frame stores the six joint positions, the six joint commands (the action the model learns to copy), the wrist camera at 640×480, the overhead camera scaled to 640×360, and one instruction for every run: "Pick up the tube by the cap and lift it." Frames are saved 30 times a second in LeRobot's dataset format.
- Keep or drop runs. Human teleop demos are used as recorded. From the scripted skill, only runs that ended with the tube lifted are kept (27 of 73 (37%)), including runs where it missed once and then recovered.
- Trim idle frames. A frame counts as idle when no joint command moved more than 0.15 joint units since the last frame. In every idle stretch the first 3 frames are kept and the rest are dropped. The pause itself is still there, but it no longer takes up half the data. This was added for v3 after v2 learned to hover.
- Convert the video. LeRobot saves video as AV1, which is slow to decode. On the GPU machine it is converted to H.264 before training, which made each step about three times faster.
- Upload. Each dataset is pushed to a private Hugging Face repo with a version tag so the training machine can download it.
| Dataset | What changed | Episodes | Frames |
|---|---|---|---|
| v1 | 20 teleop demos as recorded | 20 | |
| v2 | Added 27 successful scripted runs | 47 | 31,812 |
| v3 | Idle frames trimmed in the scripted runs | 47 | 23,936 |
Model version 0: the base model, as is
Dataset: none. Performance: not usable on this arm.
Sped up to 5 seconds.
Out of the box, the base model drives the joints toward zero, folding the wrist down and closing the gripper, then holds still. It has never seen my arm's joint ranges or my two cameras, so its outputs don't mean anything for this arm until it is fine tuned.
This was expected. A pretrained model knows what picking looks like in general, but each arm is calibrated differently, so it needs a little of my own data to map its knowledge onto my joints.
Model version 1: fine tuned on 20 human demos
Dataset: 20 teleop demos. Loss 0.009. Success rate: 0 of 3 (0%).
Dataset
The 20 demos recorded with the leader arm for the ACT post, used as recorded.
Training
The standard recipe is 20,000 training steps with batches of 64 frames. That was far too slow for the Mac, so Claude rented a GPU.
| Machine | Seconds per step | Full run | Cost per hour |
|---|---|---|---|
| Mac mini M4 (batch 8) | 5.2 | more than a day | $0 |
| Runpod A40 | 1.9 | about 10.5 h | $0.59 |
| Runpod H100, first try | 1.3 | about 7 h | $3.99 |
| Runpod H100, after the video fix | 0.44 | about 2.5 h | $3.99 |
Loss fell from 0.071 at 2,000 steps to 0.009 at 20,000. The Runpod bill for the day was $11.25, and the machine was deleted right after the model was saved.
Video
Sped up to 5 seconds.
In the third test, v1 finds the tube and moves to it, but the jaws close 1 to 2 cm off. It made 0 lifts in 3 tries. The pretrained model knows to go for the object. It just hasn't seen enough examples of the final grip on this tube.
Model version 2: adding 27 scripted demos
Dataset: 47 demos, 20 teleop and 27 scripted. Loss 0.016. Success rate: 0 of 2 (0%). How scripted demos work is in the appendix.
Dataset
Recording 20 demos by hand took an evening. To get more data, Claude used the scripted pick skill it had written earlier: the wrist camera finds the tube, the arm lines up, rolls the wrist to match the tube's angle, then grips and lifts. A collection script ran that skill over and over, recording both cameras and every joint command in the same format as the human demos. After each success the arm set the tube down at a random spot and angle for the next try.
| Scripted tries | Count | What happened |
|---|---|---|
| Kept | 27 | Ended with the tube lifted. Some include a miss and a second attempt in the same recording. |
| Missed the grip | 45 | The jaws closed beside the tube, or it slipped out on the lift. Dropped. |
| Lost the tube | 1 | The tube wasn't in the wrist camera's view. Dropped. |
| Total | 73 | About 37% success. |
The tries ran in 10 short sessions, not one long run. Each session stopped after 3 to 6 failures in a row, when the tube had been pushed somewhere the skill couldn't reach, and the tube had to be reset before the next session. The 27 kept runs plus the 20 human demos made 47 demos and 31,812 frames.
Training
v2 was trained on the 47 demos, this time on a cheaper L40 GPU, in about 6 hours for about $4. Final loss was 0.016, higher than v1, which is normal with more varied data.
Video
Sped up to 5 seconds.
v2 opens the gripper wide, lines up over the tube better than v1, then stays 2 to 4.5 cm above it and never closes. It made 0 lifts in 2 tries.
Why it stopped above the tube and never grabbed
The cause was in the data. The scripted demos spend about 55% of their frames sitting still, while the skill waits for the camera to settle between small moves. The close and lift take only the last 1.7 seconds. So the model learned that waiting above the tube is the most common thing to do. The skill also decides when to go down using its own internal checks, which the model can't see in the cameras.
Model version 3: idle frames trimmed
Dataset: the same 47 demos with idle stretches trimmed. Loss 0.014. Success rate: 0 of 30 (0%), but the gripper closed for the first time.
Dataset and training
For v3, Claude trimmed every still stretch in the scripted demos to 3 frames, cutting 31,812 frames to 23,936. v3 trained on an L40S in about 5 hours for $4.51. Final loss was 0.014.
Video
These are the last 5 seconds of a run at real speed. v3 comes down to the tube and closes the jaws, just beside it.
The improvement: the gripper closed
v2 lined up and then froze with the gripper wide open, every time. v3, trained on the same demos with the pauses cut out, went on to the next step: it came down to just above the tube and closed the gripper, which v2 never did. In 30 tries it went for the grab 3 times. It still never held on, because the jaws closed slightly beside the tube, but the model had finally learned that gripping comes after lining up.
Lesson
Too much idle data takes the model's focus away from what matters. In v2's data the short gripping motion was buried under long pauses, so the model learned to wait. Trimming the pauses, without adding any new demos, was enough to bring the grab back. Getting the grab right, from wherever the model actually ends up, needed a different kind of data.
What comes next?
Demos only ever show the ideal path, never how to recover from the model's own mistakes. The next post in this series lets the model drive, has an expert rescue skill step in when it gets stuck, and trains the next model on those corrections.
Appendix: what a scripted demo is
A human demo means someone moves the leader arm by hand while the follower arm copies it and everything is recorded. A scripted demo is the same recording, but the arm is driven by a program instead of a person. Claude wrote that program, a pick skill, and then let it collect demos.
- Find the tube. The wrist camera looks for the yellow body and red cap by color.
- Line up. The arm moves in small steps until the tube sits at the right spot in the wrist camera image.
- Match the angle. The wrist rolls until the jaws are crosswise to the tube.
- Grip and lift. The arm goes straight down, closes gently until the jaws stop on the tube, and lifts.
- Reset. It sets the tube down at a random spot and angle, and starts the next try.
Every try is recorded in exactly the same format as a human demo: both cameras, the joint positions and the commands. Only the tries that end with the tube lifted are kept. That's the appeal: demos at no human cost, from many more tube positions than a person would bother with.
| Human demo | Scripted demo | |
|---|---|---|
| Who drives | A person with the leader arm | The pick skill, a program Claude (an LLM) wrote, launched and watched. The moves come from the program, not from the LLM step by step. |
| Cost | About 20 demos in an evening | 73 tries over 10 short sessions. 27 lifted the tube and were kept (about 37%). 45 missed the grip and 1 lost the tube, and those were thrown away. Each session stopped after 3 to 6 misses in a row for a tube reset. |
| Motion | Smooth and continuous | Small moves with pauses while the camera settles |
| Watch out for | Few demos, mostly similar positions | The pauses can teach the model to hover, which happened in v2 |
The same skill later becomes the expert in the next post: when the model gets stuck, the skill takes over and finishes the pick.