Raw capture â raw/real-time-multiplayer-interfaces.md
url: https://moldandyeast.substack.com/p/real-time-multiplayer-interfaces date_fetched: 2026-07-05 backfilled: true
Version of a thesis I wrote here on content.moldandyeast.com.
Cronjobs are having a comeback. We add little heartbeats to our agents, they start to hold state (durable objects ? =)), and state is the thing that turns a tool into a participant. Once both sides persist, every interface is a multiplayer interface. Which means we need real-time ones.
By real-time I donât mean fast alone, speed will be solved. I mean a fluid dialogue, where both sides continuously listen, respond, and can interrupt. Not the trad sending a prompt and waiting through a thinking loop. Something more granular, more back and forth. A true conversation. Agents are increasingly durable now, they hold state, and self-driven, they run on heartbeats. They can start to read from sensors, APIs, cameras, the room. And simultaneously our roles are shifting underneath all of it. We move from the center, as operators and executors, to the edges, as initiators at the first mile, confirmers at the last, or observers a mile above.
The agent can cut into my thought the same way I cut into its task. So real-time collaboration hinges less on the technology and more on a shared social protocol between us and them. Both sides need to read each otherâs state and know when, how, and whether to interrupt, without fracturing attention or spiraling each other off task. TLDR: we both need to be able to read the fucking room. ;)
So far, interrupting an agent mid-task can make it spiral, the way we do when we get yanked off a thought. They are more human then we give them credit to already. Agents though will get better at holding focus (thatâs how tech works). But they are the ones who have to learn when, where, why, and especially when not to interrupt. Although it probably still cuts the other way too. The ease of running many streams at once fractures our attention. So our current obvious fix, let agents skip asking for permission and just report when done, but that only moves the problem. We add more streams and more streams and need to find the next layer to enable skipping again. Key is letting them interrupt us mid-thought makes everything worse (usually). If you think your attention is fractured, wait till you unleash the swarm.
Reading and responding cost something on both sides, but not the same thing. For the agent it costs computation, and computation is buyable. I guess thats the bull case for what all those data centers are for. For us in reality it costs attention, and attention is not buyable (only out-sourcable). We already struggle with the amount we are asked to pay. So under incomplete and shifting data, prioritization and misunderstanding become the things we are actually designing around.
We can already generate any interface for any task we can articulate. The challenge is no longer whether we can generate them (it never was if we should). It is whether they match intent and context, in the moment, as both sides read and respond to each other. So we stop designing fixed moments. We design the constraints, primitives, and protocols from which the right interaction gets ideally derived in the moment (THE DREAM).
That shift, from drawing screens to designing the system that grows them, is what I am currently explore:
Both sides observable to each other. Not a one-way adaptive surface, a loop.
Proportionate to the payoff, on each side. âAdd more computeâ and âdemand more attentionâ are not design solutions.
When, and especially when not, in both directions.
The question is not whether interfaces can be generated, but whether they match.
Grows through a two-way loop, stays self-reflective, never compounds our preferences into bias.
That last one is the hard one. The moment we co-create a shared abstraction layer, it can carry quiet bias, and it starts cold, with no precedent to draw on. An adaptive interface can only respond well to a situation we have met at least once. Everything else is a guess. So maybe we rehearse. Small, recurring, Duolingo-like interactions that surface the edge cases before the high-risk moment, not during it. You don't want to work out with an agent how to handle a security incident while it is happening.
=> How do we read and respond to each other in real-time?
None of these are solved. That is the point. Letâs play together.
This blog post was derived from my thesis over here and written by Claude in an attempt to capture my voice.
Original generation: here
I really think I have solved this. Context is actually extremely simple if you think about it symbolically. This whole angle of building from core intent is the wrong move in my opinion. Intention as much as people like to think originate within, actually doesnât informationally. Language is stupid in thinking about interfaces. The better way would be to think by relation, pattern and structure instead of semantic reasoning.
"That shift, from drawing screens to designing the system that grows them, is what I am currently explore..."
Seeing this exact same thing in my product work- we should talk ;)
My own short thoughts on the subject...
https://doingbusinessdesign.substack.com/p/ai-and-boundary-conditions-as-a-new