Design
A remote operator is driving a rover across a site. Everything works. The video is clean, the commands land, the robot does what it is told. Then, once every few hours, the operator's screen freezes. Not for long. Not in any pattern anyone can name. Then it comes back, and everything works again.
We get called into situations like this often. Teleoperation and real-time communication sit under most Physical AI deployments. The failure that causes the most problems is the intermittent one. It's just hard to understand why it happened because it can't be reliably duplicated.
Every Physical AI system runs on a chain of handoffs. The environment hands the machine a reading. The machine hands the operator a state. The operator hands the engineer a request because something broke. Most of the off-robot software we build at Andes Path sits at one of those seams. In this client case, a frozen screen (the machine-to-operator handoff) failed in a way the engineer couldn't reproduce. This post is about how we fixed that for this client, and why the tool we reached for was simulation rather than more logging.
In the video, Steve walks through the simulation environment we built. This post is the longer version: what we decided to build, how the network became part of the simulated world, and how real incidents turned into scenarios we could run on demand.
Two things to build, not one
The obvious framing of the problem is "well hey just make the teleoperation software less fragile." That's a part of it, but physical AI systems live in the real world, and the real world has this habit of changing. Software devs can't easily anticipate a rover rounding a berm into a cellular dead zone while the Wi-Fi hands off, or satellite latency spiking during a command burst, or packet loss that only appears under a specific combination of position and load.
We treated this problem as two builds.
The first is teleoperation and remote monitoring software that survives strange network conditions. The second is a simulated world that can produce those strange conditions whenever an engineer asks for them. Without the second, the first is tested by waiting. With it, the once-every-few-hours bug becomes a scenario you run 40 times before lunch.
Most robotics simulation stops at the physical world: terrain, lighting, sensors, end effectors. For a teleoperated system, a complex multi-point network is not a parameter; it's a part of the environment the robot moves through. We treated it like one.
The network is terrain
The sim we built treated the network like it was a ditch, or a pile of rocks. It is a region of the world with a position and a shape, and the rover has to navigate it. This client's rover relies on three radios: Wi-Fi, cellular and satellite. The remote operator station has one network. As the rover moves across the terrain, each radio's signal strength is read from its coverage map at the rover's current position, and that signal strength reshapes the radio's behavior: latency, available bandwidth, and packet loss all move with it.
That means the 3D environment and the network environment interact in the way they do on a real site. Drive behind a building and the Wi-Fi coverage map says the signal drops, so the simulated Wi-Fi link gets slower and lossier, and the software has to decide what to do about it. Crest a ridge and cellular comes back. Nothing in the teleoperation stack is told any of this directly. It experiences it, the way it would in the field. To keep the sim honest end-to-end, a Chrome browser runs alongside.
Three radios, three different networks
The three radios sit behind a different network configuration. Wi-Fi on a site is usually a local network. WebRTC can often connect directly, peer to peer, with the lowest latency the stack can offer. Cellular and satellite paths go through carrier-grade NAT and cannot be reached directly, so they need a relay (a TURN server, in WebRTC terms). That relay adds a hop, and the hop adds latency and a second place where the connection can be lost.
So as the rover crosses from Wi-Fi coverage into cellular, the software is seeing a different topology: a direct connection giving way to a relayed one, with the ICE negotiation, the candidate switching and the keyframe request that come with it. Satellite adds a third shape, with latency measured in hundreds of milliseconds and a bandwidth profile that punishes video.
The simulation models each of those paths with its own characteristics, and the coverage maps decide which of them is viable at any moment. A handoff between radios is a real event in the simulation, with a real cost, rather than just a number changing in a config file.
From logs to scenarios
A world that can produce bad networks is the basis. It gets powerful when we use it to encode real issues into repeatable scenarios, and the issues come from data.
For this client we sat with their team, read logs, and watched operators work. The frozen screen that happened every few hours was not one thing. It was dozens of things: a specific handoff at a specific corner of the site, a satellite latency spike coinciding with a burst of commands, a relay reconnection that the video pipeline handled badly. Each one left a trace in the logs, and each trace could be turned into a scenario: a path for the rover, a set of coverage conditions along it, and the network behavior the logs said had happened.
That gave us a library of dozens of repeatable scenarios that had the rover moving and the network operating through a simulated world that modeled both. A bug that used to appear once a shift now appeared on the third run, every time. Once we found and fixed a root cause, we took the software back to the field, watched what the real site did next, and refined the simulation with it.
That loop is the method. Field data defines the scenarios, the scenarios reproduce the failures, the fixes go back to the field, and the field tells you what the simulation is still missing. The simulation gets more faithful with every pass, and the number of "it only happens sometimes" bugs goes down. It's not magic exactly, but we've been told it feels like it to the engineers who previously didn't have a way to pull this off.
What the scenarios told us to change
With the failures reproducible, the fixes stopped being guesses. In this case most of the improvements came down to making smarter network priority decisions: what to send, what to compress, and what to drop, based on which radio the rover was on and what the robot was doing at the time.
A teleoperation link carries two kinds of traffic with opposite needs. Video is heavy and tolerates loss; a dropped frame is a glitch, not a failure. Commands are light and tolerate nothing; a lost or delayed command is a robot that did not stop when it was told to. On a good link both get through and nobody has to choose. On a satellite link in a cellular dead zone, something has to give, and the software has to be the one to decide, in the moment, with the information it has.
So the decisions became explicit. Video gets compressed harder or dropped to a lower rate as the available bandwidth falls, rather than contending with commands for the same bytes. Command traffic gets priority and its own treatment on the lossy paths. The behavior differs by radio, because the three paths fail differently, and by what the robot is doing, because the right trade during a precision maneuver is not the right trade during a transit.
None of those rules could have been written confidently from logs alone. They were written against scenarios that reproduced the failure, and they were checked against the same scenarios until the screen stopped freezing.
Why this transfers
We have built a lot of simulation systems, and the next one will be different, because the behavior worth reproducing is different for every robot. For this client it was a rover crossing between three radios. For another it might be a warehouse fleet losing Wi-Fi in a steel-racked aisle, a surgical system on a hospital network with its own firewall rules, or a drone whose link degrades with altitude. The radios change. The site changes. The method does not.
The method is: decide what part of the real world your software actually depends on, then model that part as faithfully as the terrain. For teleoperation the answer is the network, and the network has a geography, so it belongs in the world with everything else. Then let field data write the scenarios, so what you reproduce is what actually happened rather than what you imagined might.
The handoff this project sits on, from the machine to the operator, is the one a person feels most directly. When it fails, the robot is fine and the operator is staring at a frozen frame. Watch the video to see the simulated world in motion. And if you are at a Physical AI startup or a research lab and want more of your hard problems to be reproducible, in 3D, on the network, or wherever the strange behavior lives, that is the kind of simulation we build alongside robotics teams. We should talk.
