Mind-Machine Sync: Could Brainwaves Be the Secret to Advanced Physical AI?

The frontier of physical AI is a Jenga game in a warehouse in San Leandro, California.

That warehouse is occupied by Encord, a company that builds data tooling used to train AI models. Andrew Ceja is a pilot—the company’s term for its robotic trainers—and he’s carefully pulling wooden blocks from a tottering tower while wearing a headset with a camera that tracks what he sees. That alone is fairly common for collecting robot training data, but this headset includes sensors that measure his brain waves as he carefully disassembles the block tower.

Key Takeaways

• Physical AI’s Data Dilemma: Unlike text-based generative AI, training physical robots is bottlenecked by the sheer scarcity and complexity of real-world, high-fidelity interaction data, requiring innovative generation methods.
• Beyond Annotation: Manufacturing Data: Companies like Encord are moving beyond simply managing existing data to actively ‘manufacturing’ specialized datasets, including novel modalities like brainwave and muscle activity, to deduce intent and error in human operations.
• High Cost, High Value: While significantly more expensive to produce than web-scraped text data, densely annotated physical training data is proving exponentially more valuable for teaching robots nuanced manipulation skills, fundamentally altering the economics of embodied AI development.

Mind Over Machine: Unlocking Robotic Dexterity with Brainwave Data and Beyond

In a San Leandro warehouse, the future of embodied artificial intelligence isn’t unfolding in a sterile lab, but through the delicate movements of a human “pilot” playing Jenga. Andrew Ceja, an Encord robotic trainer, isn’t just manipulating wooden blocks; he’s creating critical training data for robots, augmented by a camera-equipped headset that also records his brainwave activity. This innovative blend of human action and neuroscientific insight is at the heart of Encord’s mission to crack one of the toughest nuts in AI: teaching robots how to interact with our complex physical world.

Encord, a data tooling company, is on a quest to solve the burgeoning data scarcity problem for humanoid and warehouse robotics. While model architectures continue to advance, the true bottleneck, many now believe, is the sheer lack of high-quality, real-world physical training data. Rather than merely helping companies manage the data they possess, Encord is pioneering a business model centered around *manufacturing* the data that simply doesn’t exist yet.

The Data Bottleneck: A Generative AI Conundrum

The success of large language models (LLMs) has ignited the promise of “generative AI for robots.” Yet, this vision consistently collides with a formidable wall: data. While LLMs gorged on the vast, freely available textual data of the internet, equipping a robot with equivalent physical understanding demands a fundamentally different approach. Self-driving car companies, for instance, collect their own physical-world data, but this process is inherently difficult to scale. Training from video offers a partial solution, but it often lacks the intricate fidelity required for precise physical manipulation.

Vineeth Velmurugan, Encord’s head of robot learning and a veteran of OpenAI’s robot lab, describes this challenge as the “bleeding edge” of robotics. He estimates that to achieve a significant breakthrough, physical AI models might require data sets five times the size of YouTube’s entire video corpus. Such an astronomical scale underscores why data generation itself has evolved from a research problem into a burgeoning industry. Encord, initially focused on annotating data for machine-vision applications, quickly realized that as its robotics customers embraced end-to-end learning for manipulation tasks, they needed to produce, not just manage, this elusive data themselves.

Encord’s Innovative Approach: Manufacturing Understanding

Robotics companies are increasingly looking to two primary data sources: “egocentric” video captured by workers wearing cameras, often supplemented with additional camera angles and metrics; and data generated by robots operated remotely. Encord utilizes both, drawing egocentric data from factories worldwide and leveraging its San Leandro facility as a hub for experimentation with novel modalities and the collection of data sets for fine-tuning specific skills.

Brainwaves, Body Signals, and Human Intent

The brainwave headset worn by Andrew Ceja is a prime example of Encord’s innovative data generation. Developed by Zander Labs, a German neuroscience startup, the headset measures brain activity to deduce mental states such as error, intent, and surprise. The trial run aims to create an initial brainwave-tagged dataset, test its impact on customer robotics models, and evaluate its performance. Lucas Gehrke, a Zander neuroscientist, explains that the amount of brain activity during a task can offer crucial clues to model builders, indicating when to deploy their most complex and computationally intensive models.

Beyond brainwaves, Encord is also developing forearm sensors that detect electrical signals in muscles. Traditional video of human hand manipulation often fails to capture the full dexterity. Velmurugan hopes these arm sensors can build a 3D depiction of the hand’s position and movement at any given moment, providing a richer, more robust understanding for robot learning models.

The San Leandro Lab: A Microcosm of Future Automation

A visit to Encord’s San Leandro facility reveals a fascinating blend of human ingenuity and robotic ambition. Pilots operate “leader-follower rigs”—paired robotic arms where one is directly controlled by a human and the other mimics its movements. I observed tasks ranging from pouring coffee (a surprisingly sloshy affair) to stacking poker chips, with Velmurugan noting, “Every humanoid company has asked us for these pieces.” Storage racks brimmed with props designed to train manipulators for household tasks: fake flowers, books, plastic vegetables, kitty litter trays, and bundles of wires.

At one station, pilot Sofia Infante skillfully maneuvered robotic arms to plug and unplug ethernet cables from a server – a task data center operators would eagerly automate. Taking a turn at the controls myself, the challenge became immediately apparent: robotic pincers lack the dexterity and degrees of freedom that human fingers and arms take for granted, highlighting why such “simple” tasks remain stubbornly out of reach for current automation.

The Economics of Embodied AI Data: A Cost-Benefit Analysis

Encord’s generated data sets are meticulously annotated with physical descriptions, such as “right hand tightens bolt,” to help LLM-based models understand the precise actions occurring. Velmurugan estimates that this kind of dense, high-quality annotation is exponentially more valuable—perhaps 100 times—than “junky ego data” for training specific tasks. Crucially, he calculates it only costs 20 times more to produce, presenting a compelling trade-off on paper.

However, that “20 times more” is the critical distinction. The foundational models of text-based generative AI were built by scraping the internet for next to nothing. Generating physical training data, by contrast, demands significant investment in specialized hardware, facilities, and a skilled human workforce. This fundamental difference in cost completely reshapes the economics of building physical AI models, moving from free collection to expensive, deliberate manufacturing.

Encord’s Strategic Vantage Point

Encord’s unique position, working with numerous leading robotics firms, provides an invaluable vantage point. Velmurugan sees firsthand which data techniques and modalities are gaining traction across the industry, offering insights that no single customer could achieve independently. This ability to spot emerging trends and best practices gives Encord a strategic edge in guiding the development of physical AI.

The dozen or so pilots at Encord’s facility, including Infante and Ceja (both formerly of AI data annotation firm Scale), are at the forefront of this burgeoning field. Ceja, whose interest in technology began keeping a robotic trash sorter running at a waste management company, finds deep satisfaction in the daily challenges of training robots. “It’s something new every day!” he remarks, as the Jenga tower inevitably topples.

Bottom Line

The future of robotics hinges not just on smarter algorithms, but on the ability to generate and refine vast quantities of real-world interaction data. Companies like Encord, by pioneering the ‘manufacturing’ of high-fidelity training datasets—even incorporating groundbreaking modalities like brainwave activity—are directly addressing the fundamental bottleneck preventing physical AI from realizing its full potential. While the cost of this specialized data generation is high, its value in unlocking truly dexterous and intelligent robot behavior is proving indispensable, fundamentally altering the development roadmap for embodied artificial intelligence.