Design
An autonomy engineer wants two things from a robot's data. To see what the machine is seeing right now, while it is out there doing the job. And to see exactly what it saw last Tuesday at 14:07, frame for frame, with every sensor lined up, so they can find out why it did what it did. Both of those sound like playback. In practice they are two different systems, and most teams build them twice.
The client in this case runs a remote application whose details stay confidential, but the shape of the data is the shape of most robots: video, IMU, pressure, and a handful of other streams arriving at different rates. Their engineers needed live visualization during operation, and they needed to record everything and replay it later for CI, for testing, and for an ML pipeline that had its own appetite for synchronized data.
The video here is Clark Jeria, our co-founder, walking through the engineering decisions behind what we built. This post is the longer version, because the decision at the center of it is one we think more robotics teams should be making.
Robot data is heavier than video
Streaming video with synced audio is a solved problem, even at Netflix-level scale. Robot telemetry is heavier, arrives from many sources at once, and the synchronization between those sources is the whole point. A pressure spike is meaningless unless you can see the IMU and the camera frame from the same instant.
The conventional approach, and one we have used before, puts the responsibility for synchronization, buffering, and replay logic in the client. That works, up to a point. The point is usually when the client is a browser and the throughput is a real fleet. Browsers are simply not built to hold, align, and scrub through multiple high-rate sensor streams at once, so the client grows a large and fragile block of replay logic, and the engineer needs a very powerful machine to run it.
Put the replay logic where the compute is
This time we moved all of it to the backend. The client asks for the latest frame it should show for each stream. It does no synchronization and no data handling of its own. Everything else, including alignment across streams, buffering, seek, and playback speed, happens server-side.
Doing that shifts the hard problem rather than removing it, because now every scrub, seek, or speed change is a command to a remote system that has to respond fast enough to feel like local playback. The answer was to give every replay session a unique identifier. When the client seeks, that action goes straight to the session's stream, which already knows its state and can rehydrate from preloaded data instead of dropping what is in memory, cleaning up, and loading again from scratch. The client sees an updated stream almost immediately.
The other thing the backend move buys is real capability. True multiprocessing and multithreading, machines sized for the data, and the freedom to optimize the pipeline without worrying about what a laptop browser tab can sustain.
What a headless platform is worth to an engineering team
The client swapped front ends several times over the course of the project. Each time, the change cost almost nothing, because the front end was only ever asking for frames. When they later decided to tie the whole thing into Foxglove, that came with no additional engineering effort on the replay side.
All of you engineering leaders just reread that last bit, and probably smirked a little. You know a headless replay platform means live visualization, CI replay, test harnesses, and the ML pipeline all consume the same synchronized stream from the same source of truth. When a new front end shows up, it just plugs in.
First principles, on a deadline
Clark is candid in the video that we have built systems like this before and that they get tricky in proportion to the data. The reason to try something different was not novelty for its own sake. It was that the previous approach had a ceiling the client was going to hit.
That is the balance we try to hold on every Data and Simulation project. Solve the immediate need quickly, because the team has robots in the field and cannot wait. In parallel, do the real technical design, and be willing to suggest the less obvious architecture when it is the one that will still be standing at ten times the scale. The systems we are proudest of are the ones a client never has to rewrite because of their own growth.
Where it landed
Today, the client's engineers see live sensor data as the robots produce it and replay any recorded session with every stream aligned, from the same platform, through whichever front end they choose. The same recordings feed CI, testing, and the ML pipeline. The replay logic lives in one place and scales with the infrastructure rather than with the browser.
Watch the video to hear Clark walk through the design in his own words. And if your team is spending its time keeping sensor streams in sync instead of learning from them, that is the kind of Data and Simulation infrastructure we build alongside robotics teams. Let's talk.
