Robot Model Experiment Series 1: Opus 5.5 + Robot Skills

Claude Opus 5.5 can control a robot arm, but not the way most people picture it. It is too slow to steer the arm live, so it works like an engineer instead. It looks through the cameras, tries a move, checks what happened, and then writes a program that does the job without it.

This is the first post in a series where I run different robot models on the same small task: picking up a lip balm tube from my desk. Here the "model" is Claude itself, with no robot training at all.

What does the setup look like?

The arm is a Hiwonder SO-101, a small open source arm with six motors, one of them for the gripper. It plugs into a Mac mini over USB, along with two cameras. One camera sits on the gripper and sees the tube up close. The other is a cheap webcam on a boom stand that looks down at the whole table. A desk lamp keeps the light steady.

The black arm in the corner is a second SO-101 called the leader. I only use it when I want to drive the arm by hand, which matters for the next post in the series.

Why can't Claude steer the arm live?

A robot arm needs fresh commands about 30 times a second. Claude takes a few seconds to look at a camera frame and decide what to do. That is fine for planning and far too slow for steering.

So the first run was slow on purpose. Claude looked at a photo, moved one joint, and looked again, about 15 times, and the pick took 120 seconds.

Sped up to 5 seconds.

How did it get from 120 seconds to 15.6?

After the first run, Claude wrote down what had worked and turned it into a program. The program finds the tube by its color, moves the gripper to it, nudges the arm until the tube sits between the jaws in the wrist camera, and closes the gripper until the jaws stop moving.

Then I added the overhead camera. Claude calibrated it by itself, placing the tube at known spots and recording where the camera saw it. After that the arm could go straight to the tube instead of searching for it.

Sped up to 5 seconds.

What robot skills did Claude actually show?

The list surprised me, because most of it is the unglamorous work a robotics engineer does.

• Bring up the hardware. It found the USB ports, scanned all six motors on each arm, read their voltage limits and calibrated both arms.
• See. It found the tube by color in both cameras and retuned the colors when the room light changed. It also built a live focus meter when the wrist camera was blurry.
• Move. It wrote a small geometry model, so it could say "put the gripper 15 cm out and 3 cm above the table" instead of six joint angles.
• Grip gently. It closes until the jaws stop moving instead of squeezing, which stopped the motors from overloading.
• Handle data. When I recorded demos for the next experiment, it fixed the recorder, cut long recordings into single demos, and set aside the bad ones.

Where did it still need a person?

Three kinds of things. The first is hands on fixes, like taking the lens cap off the camera, plugging a cable back in, or moving the tube back within reach.

The second is physical know how. Two of my tips fixed the grip. Aim for the upper quarter of the tube near the cap, because the middle is thin and slippery, and keep the jaws at a right angle to the tube or it slides sideways. Claude could not have learned either from the camera alone.

The third is demonstrations. If you want a robot to learn by copying, a person has to do the task first. Claude can record, check and clean that data, but it cannot produce human motion.

Could the skill run 50 times on its own?

The first runs had me nearby, placing the tube and fixing things. Over the next few days the program grew into the expert rescue skill that finishes the job whenever a trained model gets stuck later in this series. So I gave it a harder task: 50 runs in a row with no person at all.

• The arm starts each run from the same resting pose.
• After each run it sets the tube down at a random spot: 13 to 21 cm out, up to 35° to either side, turned up to 45°.
• No overhead camera. Its calibration had gone stale, and the arm blocks its view at the start pose, so the skill finds the tube with the wrist camera alone.

Along the way the skill changed in ways that came from watching its failures:

• It looks from where it is instead of going back to a home spot, and moves in one continuous motion.
• It rolls the jaws across the tube and never closes unless the tube is lined up between them.
• It lowers until the motors feel the table, rises 6 mm, then closes.
• It squeezes a little past the point where the jaws touch, and aims nearer the cap, where the tube is round.
• If the first grab misses, it tries once more from where it is.

The wrist camera during a 13.7 second pick, sped up to 5 seconds. The jaws line up across the tube before they close.

How did the skill improve over time?

The 50-run time is the median of the 41 picks that worked on the first try. Four more worked on the second try, in 26 to 42 seconds.

The big gain is reliability. It went from a few supervised runs to 50 unattended ones, with the tube anywhere and at any angle. Lighting mattered: the first 14 runs were at night and picked 10, and the 36 in daylight picked 35.

The typical pick did not get faster. The median of 18.4 seconds is a little slower than 15.6. The skill no longer has the overhead camera to go straight to the tube, so it searches with the wrist camera. Lowering until it feels the table and closing gently each add a second or two, and the clock now runs until the tube is lifted and the grip is checked. Its fastest pick, 12.2 seconds, beats every earlier run.

How is this different from training a robot model?

Claude's program reached the tube anywhere on the table with zero demos. But it only works for this tube and this motion, and every change means rewriting code. It is closer to a very fast engineer than to a robot that has learned a skill.

The next post trains a real robot model on demonstrations I recorded, and it fails in the opposite way. If you want the background on how robots learn from people, I wrote about that in how robots learn from us.

The short version

Claude Opus 5.5 cannot steer a robot arm in real time, but it can set one up, see with its cameras, and write a program that picks up a tube in 15.6 seconds without a single demo. It still needed me for physical fixes and for know how about grip. Refined over a few days, the same program picked the tube in 45 of 50 runs with no person, from random spots and angles. A general skill that works on new objects still has to come from data and trained models, which is where this series goes next.