The big trend in robots is handing the keys over to a generative AI model, but that brings with it a problem: that architecture isn’t predictable the way traditional algorithms are. How can you be sure your brand new humanoid will be safe?
Dr. Ding Zhao, who directs the Safe AI lab at Carnegie Melon University, has been working on this problem for almost his entire career. Now, along with veteran start-up executive Kyle Wong and machine learning engineer Simo Rachidi, he’s founded a company, Safeworld, intended to solve it.
“The safety challenge that we’re talking about is a combination of, one, really advanced generative AI probabilistic evals — how do you underwrite the risk of a probabilistic system?” Zhao says. “The second part that’s really hard is the trust part, and you need both to deploy a robot.”
Safeworld is emerging from stealth today with a seed round of more than $12 million, led by Shine Capital and a16z Speedrun, with additional investment from Box Group, Carnegie Mellon University Endowment, Innovation Endeavors and SV Angel.
“The time to build an industry safety standard is now while robots are being designed and deployed,” a16z Speedrun partner Jonathan Lai told TechCurnch. “By the time you have robots in households colliding with kids and causing safety incidents, that’s way too late.”
Safeworld’s speciality is evaluating a robotic control system in simulations that are populated with realistic human models. It’s akin to the challenge faced by companies like Tesla or Wayve, who must ensure that their vehicles respond appropriately to a variety of surprising incidents they may encounter on the road. But that will be more difficult for robots, Zhao argues, because they work in unstructured environments, and because each facility they are in will have different safety standards.
“One of the most common areas is if there is a blind corner in this particular factory,” Wong said. “What is the speed or what is the stopping distance that you need to make sure that this robot will not collide with a particular human? If a human is carrying boxes, for example, will the robot detect the human or not?”
To answer that question, Safeworld will build a digital version of that corner in a model like Genesis or MuJoCo, insert a simulation of the robot it is evaluating, driven by its real software, and then run thousands of scenarios where human models encounter the robot. That’s harder than it seems, per Zhao, because people are unpredictable.
“Tripping and falling is also a good example of something that we do a lot of testing with the simulation,” Wong said. “Otherwise, you would have to go and trip and fall for the robot, which is like a hard thing to be doing all the time.”
There are definite similarities between the platform that Safeworld is building and the tools being used internally by robot builders. The founders, however, believe that beyond their specific expertise, robot-makers will want a third-party to validate their work, if only to share information about safety cases between competitors.
“A lot of people are underestimating one how hard some of these edge cases are going to be to solve,” Zhao said. “It is not the robot in the vacuum, in the demo, that we are worried about. It is the robot that is deployed at scale, with people who potentially never operated a robot before.”
Vishal Dugar, the CTO of Gritt Robotics, is developing the AI brain for robots that currently help workers install photovoltaic panels at industrial-scale solar farms, and aspire to take on more complex construction tasks. His company is partnering with Safeworld as they develop their safety simulations.
“The difficulty with most of our systems is it’s very hard to formally prove it by doing some math, writing some equations, and saying yeah, the system is verified to be safe,” Dugar says. “It necessarily has to be done empirically.”
His robots operate alongside human workers, and ensuring that its robotic arm doesn’t hit them is obviously top of mind. To verify that in practice will requires considering all kinds of potential scenarios.
“Humans have many kinds of appearances,” Dugar points out. “Their bodies can be in different configurations. They could be kneeling, standing. They could be tripping and falling potentially. They could be crouching. They could be running. You have to respond to all these behaviors that humans could potentially exhibit on these sites, along with the variety of variations in human appearance, you know, clothes, size, shape, height, skin color, everything else.”
It’s still early days for both Safeworld and generative AI in robotics, and the company is still figuring out the best model for its product—a platform for external users, or a services based approach?—but the team is confident they are taking on the right problem.
“We’ll probably be the first profitable company in this field,” Zaho says. “Because if anyone wants to deploy, they need to pay us to handle the situation.”
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
Safeguarding the Rise of AI Robots: Inside Safeworld’s Mission to Prevent Disaster
Key Takeaways:
- Generative AI’s Double-Edged Sword in Robotics:While generative AI promises unprecedented capabilities for robots, it introduces a critical unpredictability challenge, making traditional safety protocols insufficient for human-robot interaction.
- Safeworld’s Simulation-First Approach:Backed by over $12 million in seed funding, Safeworld is tackling this by pioneering advanced simulation environments that stress-test robotic control systems against realistic and unpredictable human behaviors and complex edge cases.
- Establishing a Crucial Third-Party Standard:The company aims to become the indispensable third-party validator for robot safety, fostering industry-wide trust and enabling the widespread, secure deployment of AI-powered robots in diverse, unstructured environments.
The dawn of generative AI has ushered in an era of remarkable innovation, with its influence now profoundly reshaping the field of robotics. From industrial automation to household assistance, the vision of intelligent machines seamlessly integrated into our daily lives is rapidly transitioning from science fiction to imminent reality. Yet, this transformative power comes with a fundamental caveat: the inherent unpredictability of generative AI models. Unlike traditional, rule-based algorithms, these probabilistic systems operate with a degree of autonomy that makes their behavior challenging to forecast and, critically, to guarantee as safe, especially when interacting with humans. This burgeoning dilemma poses a monumental question: how do we ensure the safety of our brand new, AI-driven robotic counterparts?
For Dr. Ding Zhao, who directs the Safe AI lab at Carnegie Mellon University, this isn’t a new concern but a lifelong pursuit. Now, alongside seasoned startup executive Kyle Wong and accomplished machine learning engineer Simo Rachidi, Dr. Zhao has co-founded Safeworld, a pioneering venture dedicated to resolving this very problem. Safeworld’s mission is clear: to establish robust, verifiable safety standards for the next generation of AI-powered robots, ensuring their secure deployment in a world increasingly shared with humans.
The Unpredictability Problem: Underwriting Risk in a Probabilistic World
The core challenge, as Dr. Zhao articulates, is two-fold. “The safety challenge that we’re talking about is a combination of, one, really advanced generative AI probabilistic evals — how do you underwrite the risk of a probabilistic system?” he explains. “The second part that’s really hard is the trust part, and you need both to deploy a robot.” The shift from deterministic, programmed responses to probabilistic, AI-generated actions means that the traditional methods of proving a robot’s safety — mathematical formalisms and rigid logical proofs — often fall short. This new paradigm necessitates a radical rethinking of how we validate safety, moving towards empirical testing and comprehensive risk assessment for systems that learn and adapt.
This urgent need for a new safety framework has not gone unnoticed by the venture capital community. Safeworld is emerging from stealth today, having successfully secured a robust seed round exceeding $12 million. The funding round was spearheaded by prominent investors Shine Capital and a16z Speedrun, with significant additional contributions from Box Group, the Carnegie Mellon University Endowment, Innovation Endeavors, and SV Angel. This substantial backing underscores the market’s recognition of the critical void Safeworld aims to fill.
Safeworld’s Digital Sandbox: Simulating Safety at Scale
Safeworld’s innovative approach centers on evaluating robotic control systems within highly realistic, dynamic simulations populated by sophisticated human models. This methodology shares conceptual similarities with the rigorous testing undertaken by autonomous vehicle developers like Tesla or Wayve, who must ensure their cars can navigate unpredictable road scenarios. However, the challenge for general-purpose robots is arguably far more complex. Dr. Zhao highlights a key distinction: robots operate not just on predefined roads but in “unstructured environments,” where every factory floor, hospital ward, or public space presents unique layouts, obstacles, and, crucially, varying safety standards.
Consider a scenario within a factory setting, as Kyle Wong illustrates: “One of the most common areas is if there is a blind corner in this particular factory. What is the speed or what is the stopping distance that you need to make sure that this robot will not collide with a particular human? If a human is carrying boxes, for example, will the robot detect the human or not?” To address such specific, yet infinitely variable, situations, Safeworld constructs a precise digital replica of the environment using simulation platforms like Genesis or MuJoCo. Into this digital twin, they integrate a high-fidelity simulation of the robot, driven by its actual software. Then, thousands upon thousands of diverse scenarios are executed, pitting human models – designed to exhibit a full spectrum of unpredictable behaviors – against the robot. As Wong emphasizes, testing for edge cases like “tripping and falling is also a good example of something that we do a lot of testing with the simulation. Otherwise, you would have to go and trip and fall for the robot, which is like a hard thing to be doing all the time.” The sheer scale and variability of human actions make real-world empirical testing impractical, if not impossible, for comprehensive safety validation.
The Imperative for Third-Party Validation and Industry Standards
While robot manufacturers undoubtedly employ internal testing tools, Safeworld’s founders contend that their specific expertise, combined with their position as an independent third party, is indispensable. Jonathan Lai, a partner at a16z Speedrun, echoes this sentiment: “The time to build an industry safety standard is now while robots are being designed and deployed. By the time you have robots in households colliding with kids and causing safety incidents, that’s way too late.” The cost of inaction, both in terms of potential harm and the inevitable public backlash, is simply too high. Furthermore, a neutral third-party validator can facilitate the sharing of critical safety case data and best practices across competitors, fostering a safer ecosystem for the entire industry rather than individual, siloed efforts.
Dr. Zhao underscores the gravity of the challenge, stating, “A lot of people are underestimating one how hard some of these edge cases are going to be to solve. It is not the robot in the vacuum, in the demo, that we are worried about. It is the robot that is deployed at scale, with people who potentially never operated a robot before.” The sheer diversity of human interaction and the potential for novel, unforeseen incidents mandate a level of scrutiny that goes far beyond controlled laboratory environments.
Real-World Application: Partnering for a Safer Future
The practical necessity of Safeworld’s offering is already evident in its partnerships. Vishal Dugar, CTO of Gritt Robotics, is developing AI brains for robots that assist human workers in installing photovoltaic panels at large-scale solar farms, with ambitions for more complex construction tasks. Gritt Robotics has partnered with Safeworld to develop its safety simulations. Dugar acknowledges the inherent difficulty: “The difficulty with most of our systems is it’s very hard to formally prove it by doing some math, writing some equations, and saying yeah, the system is verified to be safe. It necessarily has to be done empirically.”
For Gritt Robotics, whose machines operate in close proximity to human laborers, preventing collisions is paramount. This requires accounting for an astonishing array of human variables. Dugar elaborates: “Humans have many kinds of appearances. Their bodies can be in different configurations. They could be kneeling, standing. They could be tripping and falling potentially. They could be crouching. They could be running. You have to respond to all these behaviors that humans could potentially exhibit on these sites, along with the variety of variations in human appearance, you know, clothes, size, shape, height, skin color, everything else.” This comprehensive consideration of human diversity is precisely where Safeworld’s advanced simulation capabilities prove invaluable.
The Bottom Line
While both Safeworld and the broader application of generative AI in robotics are still in their nascent stages, the company’s founders are unwavering in their conviction that they are addressing an absolutely foundational problem. The exact commercial model—whether a platform-as-a-service or a more bespoke, services-based approach—is still being refined. However, the underlying demand is undeniable. As Dr. Zhao confidently asserts, “We’ll probably be the first profitable company in this field. Because if anyone wants to deploy, they need to pay us to handle the situation.” The widespread integration of AI-powered robots into our industries, homes, and public spaces hinges entirely on public trust and demonstrable safety. Safeworld isn’t just building a company; it’s laying the critical groundwork for a future where intelligent machines can thrive alongside humanity, not at its peril. Without a robust, independent safety net, the true potential of the AI robot revolution will remain tethered by uncertainty and fear. Safeworld aims to cut that tether, enabling innovation while guaranteeing peace of mind.
Source:{feed_title}