Designing for Physical AI Systems

Walk onto the floor of a factory that runs on autonomy and you will see the impacts of the software long before you see an actual screen. It is in how an operator in work gloves leans over a shared kiosk between tasks, or a coordinator holding two radios and a tablet, and a driver who left cell coverage 40 minutes ago and still has to prove a delivery happened.

That floor is where this project started. The client ran factory operations and dispatch across three countries with ten different tools, built by different teams in different places over several years. Each tool made sense to the team that shipped it. None of them had been designed around the people standing there in gloves, and the seams between them had become friction.

The video here features our lead designer, one of our engineers, and one of our co-founders talking through what we did about it.

Start with the process, then the people

The first weeks of a project are spent understanding how the work actually happens. Who does it. Where they stand while they do it. What is on their hands, what is in their ears, what the light is like, what the network is like. We map the whole ecosystem of people and machines before anyone opens a design tool.

__wf_reserved_inherit

This sounds obvious and it is routinely skipped, usually because the software was scoped from a feature list rather than from the floor. A feature list will tell you that dispatch needs a proof-of-delivery step. It will not tell you that the truck has no signal when the step happens, or that the person completing it is wearing gloves, or that one region has harsh winters, or that the kiosk they will return to is shared by three shifts.

‍

One blueprint, end to end

With the physical process understood, design and systems engineering work together on a single service blueprint. For this client that meant dispatch, end to end: five phases, from a customer placing an order down through the machines and the APIs underneath them. Every role, every handoff, every point where a human decision meets an automated one, on one document that both disciplines own.

The blueprint does the job that ten separate specs never could. It exposes where the same task was being represented three different ways, where a status lived in one system and was retyped into another, and where the physical constraint of the site was going to shape the architecture. Only after the blueprint is agreed do we start drawing screens, and by then most of the hard decisions have already been made.

‍

Build the foundation once

Ten applications became one platform. One shell, one design language, one set of components, with role-based views on top. An administrator sees everything. A coordinator sees the day's work. A driver sees a phone app with exactly what the route requires.

__wf_reserved_inherit

For an engineering leader, the consequence is what happens next. A new module inherits the platform instead of negotiating with it. Authentication, permissions, navigation, data conventions, and visual language are already settled, so the team building the next capability spends its time on the capability. The design system stops being a document and becomes the thing everyone builds inside.

‍

Let the site make the decision

Some delivery locations for this client have no coverage at all. That fact came out of the network map we built during the first phase, and it decided the architecture of the driver app before a single screen existed. The app is offline first. Proof of delivery, photos, signatures, all of it is captured locally on the phone and synced when the truck finds a signal again.

__wf_reserved_inherit

Offline first is a commitment, not a setting. It means the phone holds the source of truth for a period of time, the sync has to be idempotent and tolerant of partial uploads, and the interface has to make it obvious to the driver what has been captured and what is still waiting. Those are engineering problems and design problems at the same time, which is why we run them as one team.

‍

Why we work this way on Physical AI

Our clients build the autonomy core. Our work is everything around it: the operations layer that runs the fleet in production, the data and simulation tooling that validates it, the developer tooling that ships it. That off-robot software has one job, which is to run a physical process reliably with real people in the loop.

You cannot design that from a screen inward. The machine, the site, the network, and the person holding the device are the inputs, and the interface is the output. Learn the physical process first, build the foundation once, and everything added afterwards inherits it.

‍

Where it landed

Today that platform runs the work in three countries, across two products, covering factory operations and dispatch, under one design system, managing thousands of tasks a day. The people on the floor use one tool. The teams building on it start from the platform instead of from scratch.

__wf_reserved_inherit

Watch the video to hear the team walk through it in their own words. And if your autonomy is working and the software around it is where the friction lives, that floor walk is usually where the conversation with us begins.

‍

Insights

Thinking from the frontier.