Getting Physical with AI (2): A System 1 Brain for My Musubi-Making Agent

Duo tone styled image of SO-101 reaching to make spam musubi with LeRobot and GR00T logos

I made a YouTube video covering most of what’s in this post, complete with clips of the robot in action. If you’d rather watch than read, check out the video here:

My musubi agent already has a body, the SO-101 robotic arm I built in my last post. Now the challenge is to build the brain that will bring it to life so it can assemble my favorite snack.

Before we get into the details of data collection and training, let’s talk about the title of this post and what System 1 refers to.

Daniel Kahneman’s Thinking, Fast and Slow popularized a nice framework for how we think, breaking it down into two modes. “System 1” is our fast, instinctive, reflex-like thinking. “System 2” is our slower, deliberate, logical reasoning. For a robotics agent, generating the continuous, low-level motor commands to physically move an arm is the equivalent of System 1. Later on in this project, I’ll be running experiments on giving my agent a System 2 brain for higher-level planning and reasoning.

Teleoperation and Building the Dataset

To teach the robot its System 1 reflexes, I needed to show it some examples of how to move. I used LeRobot, an open-source robot learning library by Hugging Face, to record my teleoperation data.

The specific task I’m covering today is picking up a slice of Spam and placing it on a bed of rice. Future videos and posts will cover the other steps.

Using LeRobot, I recorded 50 episodes of me teleoperating the arm to perform the task. For each episode, LeRobot records the following in an open format called LeRobotDataset:

  • Image data from all 3 of my cameras (a base camera, a wrist camera, and an overhead camera).
  • The current motor positions.
  • The commanded motor positions.
  • The text instruction for the task.

Besides teleop and data collection, LeRobot comes with other handy features. I also used it to visualize the dataset and replay the episodes on the actual robot to make sure the data was good for training.

Picking a Model and Training the Brain

After collecting the episodes, I needed a base model to post-train. There are plenty. of. options. out there, but I ultimately went with GR00T N1.6, an open model designed specifically for robotics by NVIDIA.

GR00T is what’s known as a Vision-Language-Action (VLA) model. Think of VLAs as cousins to the LLMs we use every day. They are pre-trained models packed with general knowledge, but instead of chatting with you, VLAs are trained to take in multi-modal instructions (text and images) and directly generate the motor controls needed to carry out those instructions.

architecture diagram for vision-language-action models
Figure 1 – Vision-Language-Action model architecture

To post-train the model, I used the fine-tuning scripts provided in the GR00T GitHub repo. Because I don’t have a powerful GPU of my own, I used NVIDIA Brev to rent one in the cloud. It took about 30 hours of training on my 50 episodes to get the model dialed in.

I probably could have gone with fewer episodes and less post-training overall, but I stuck with what the GR00T team recommended in this article.

Inference: Did It Actually Work?

To run inference and test the robot policy, I once again used scripts from the GR00T GitHub on a GPU from NVIDIA Brev.

So… without further ado, was I able to get GR00T to control the SO-101 to make me Spam musubi?

Well, you’ll have to stick around for future updates to see if I ever get all the way there. But as for picking up a slice of Spam and placing it on a bed of rice? It worked!

If you watch the video clip of this first inference, you’ll notice the movement is a bit slow and jerky. I suspect this has something to do with my rented inference server being remotely located in France.

Even with the slight stutters from latency, seeing the robot autonomously manipulate the Spam was incredible. We’ve officially taken our first real step toward a fully autonomous Spam musubi-making robot.

What’s Next?

So far, I’ve used the SO-101, Hugging Face LeRobot, NVIDIA GR00T N1.6, and NVIDIA Brev to build (a piece of) a System 1 brain. And we successfully placed some Spam!

If you’re feeling inspired to get an SO-101 of your own, check out my previous post for recommendations on getting started.

Now what about all the other tasks that go into making musubi? I’ll be exploring a few different directions. I’ll try providing tools to general purpose LLMs to act as a System 2 planner. Alternatively, I might switch from VLAs to a newer generation of robot foundation models that are more promptable (if models like π0.7, S1, or GEN-1.5 ever get released for general use).