Robbyant releases LingBot-Touch to help robot hands sense contact

Robbyant, the robotics AI company within Ant Group, released LingBot-Touch on November 10. It gives robot control systems a common way to read measurements from three kinds of fingertip sensors. The researchers kept its weights fixed on a robotic hand with an unfamiliar magnetic sensor and trained the controller for that hand separately.

Robbyant’s researchers compared tactile inputs across 26 physical tasks, including closing a zipper, separating nested cups and unscrewing a bottle cap. In one cap test, the team reported a score of 56.25% with no touch input, 62.5% with unprocessed sensor readings and 90% with readings interpreted by LingBot-Touch. The paper’s rubric gives credit for stages along the way, such as gripping and rotating the cap, loosening it completely and lifting it clear.

Three ways to measure contact

Piezoresistive sensors register a change in electrical resistance when pressed. Magnetic sensors measure changes in a magnetic field, while optical tactile sensors use an internal camera to observe how a surface deforms under contact. Their readings arrive in different formats and may come from different positions on the fingertip. LingBot-Touch processes each signal type and maps the measurement points to a common set of fingertip coordinates. Its output describes contact and how it changes over time in a form the robot’s controller can use.

The team assembled 4,983 hours of recordings for pretraining. Robbyant counts about 1.33 billion frames, each a snapshot of sensor readings. About two-thirds came from people handling objects with gloves fitted with piezoresistive or magnetic sensors. The remaining recordings came from remotely operated robot hands with optical tactile sensors. Several sources captured similar kinds of actions, such as opening, assembling and wiping. This gave the model examples of similar contacts measured by different devices.

What changed when the controller received touch data

The physical tests used two robotic hands, the Sharpa Wave and XHand1. For each of four controllers, the team trained three versions on the same set of demonstrations: 200 per task, or 500 for tasks requiring both hands. One received no tactile input, one got the raw readings, and one used features computed by LingBot-Touch. Section 5.2 of the technical report puts the mean improvements across all 26 tasks at 13.9 to 24.8 percentage points over the no-touch versions. The 90% cap score belongs to one test on a Sharpa Wave hand controlled by T-Rex. The rubric awards partial credit, so the score is not a count of trials in which the cap came all the way off.

The results differed by task. When the same controller used Sharpa Wave to pull out a book, it scored 69% with raw readings and 59% with LingBot-Touch features. The team selected physical tasks in which touch could affect the outcome. A video on the project page separately demonstrates a robot squeezing toothpaste, one example of a movement that depends on contact.

The XHand1 uses a magnetic sensor model absent from the pretraining set, where magnetic measurements had come from human gloves. The researchers kept LingBot-Touch’s weights fixed and trained the controller on demonstrations made with the XHand1. In one nested-cup task, the π0.5 controller scored 32.5% both without tactile input and with raw readings. The score rose to 72.5% when LingBot-Touch interpreted the readings.

What the release includes

Robbyant published inference code and sample data on GitHub and two sets of weights, Tiny and Small, on Hugging Face. Both model cards list the Apache 2.0 license. The physical test table describes a Tiny encoder pretrained on around 4,000 hours of data. The downloadable bundles contain the tactile encoder and its configuration. The research team trained the task controllers separately.

The examples include 30 pairs of left- and right-hand recordings for each of the two robot hand types. Connecting a different device still involves aligning the locations and measurement ranges of its sensors. For an optical sensor, it takes deformation or intensity measurements derived from the sensor image.