ACT is a simple robot model that learns directly from human demonstrations. I trained one on 20 demos of my desk arm picking up a lip balm tube, using only a Mac mini. It learned to copy my moves very closely, and then on the real table it did not pick up the tube once.
This is the second post in my robot model experiment series. The first post had Claude write a program for the same task. This time nobody writes code for the task, and the model learns it only from watching.
What is ACT?
ACT stands for Action Chunking Transformer. It looks at the camera images and the arm's joint angles, and it predicts the next few seconds of motion all at once, as a chunk of 100 small steps. Predicting a chunk instead of one step at a time makes the motion smoother and less jittery.
ACT starts with no knowledge of robots. Its only head start is an image encoder trained on everyday photos. Everything it knows about the task comes from the demos, which makes it a good baseline, because there is nothing hidden.
How did I record the demos?
I used two identical arms. I move the leader arm by hand, and the follower arm copies it 30 times a second while both cameras record. This is called teleoperation, and it is how most robot training data gets made.
I recorded five long sessions of back to back picks, about 12 minutes in total. Claude cut them into single demos by reading the gripper's motion and checking a wrist camera frame at each grasp. I kept 20 good demos, about 6.5 minutes of footage, and set aside the misses and the long searches.
Sped up to 5 seconds.
A clean demo takes about 15 seconds. Two of the 20 show a miss followed by a second grip, which teaches the model what to do when a grip fails.
Sped up to 5 seconds.
What went wrong while recording?
More than I expected. One session with six or seven good demos was lost when the wrist camera cable pulled out and the recorder crashed before saving. Claude changed the recorder so it now saves everything up to the moment a camera fails.
The wrist camera was also out of focus at first, so the first two demos were set aside. Small hardware problems like these are a big part of collecting robot data, and none of them show up in the final dataset.
How long did training take on a Mac mini?
About nine hours, for free. The Mac mini's built in GPU ran 30,000 training steps at about one step per second. The loss, which measures how far the model's moves are from mine, fell steadily.
| Training step | Loss |
|---|---|
| 200 | 7.1 |
| 5,000 | 0.44 |
| 10,000 | 0.20 |
| 20,000 | 0.12 |
| 30,000 | 0.09 |
Did the ACT model actually learn the task?
On my demos, yes. When Claude fed it frames from the demos, its first predicted move was within about 2 degrees per joint of what I actually did. With the cameras blacked out, the error rose to almost 9 degrees, so the model really was using its eyes.
On the real table, no. In three tests it hovered over the table, opened and closed the gripper, and never went down to the tube.
Sped up to 5 seconds.
Why did the ACT model fail on the real table?
The tube was somewhere my demos never covered. In the first test it lay sideways right under the gripper, which needs a big wrist turn that only a few demos showed. In the third it sat off to the side, closer to me, and the wrist camera never saw it at all.
On the left is a typical demo start, with the tube just past the gripper tip. On the right is a test. Twenty demos taught the model one area of the table, the way a student who memorized one page of examples gets stuck on a new kind of question.
What does this mean for robot data?
A low training loss only means the model copies the demos well. The real test is a tube in a new place, and that comes down to coverage. Twenty demos spread across many positions and angles would probably beat fifty in one spot.
This is the same gap I wrote about in free robot data is not the data you need. Pretrained robot models and large, varied teleop datasets exist to close it. The next post in this series fine tunes SmolVLA, a model pretrained on thousands of community demos, on these same 20 demos.
The short version
ACT learned my 20 demos almost perfectly and still failed whenever the tube was somewhere new. The model was fine and the data was too narrow. Next I will test whether a pretrained model does better with the same small dataset.