second-brain

agent-first-software

10 min read

Agent-First Software

The design principle that software must be built primarily for agent consumption — because with 100–1000x more agents than people interacting with systems, agent experience determines business performance.

Last updated: 2026-04-23

Overview

For most of software history, every design decision optimized for the human interface: the UI, the onboarding flow, the discoverability of features. As AI agents become the dominant consumers of software systems — querying APIs, executing workflows, orchestrating data across tools — this assumption inverts. The question shifts from “how do a human navigate this?” to “can an agent effectively use this at scale?”

The core thesis, from Aaron Levie (Box CEO) and others building at this layer: if you have a hundred or a thousand times more agents than people operating on software, then your software’s design must be agent-first. Business performance will correlate with how well agents can access the information they need to do their work.

Why It’s Not Just “Build Good APIs”

The naive version of agent-first software is “expose clean APIs with good documentation.” This is almost exactly wrong.

Agents don’t navigate software the way humans do — they don’t care about documentation quality, aesthetic interface design, or onboarding UX. Instead they select tools based on:

  • Semantics: does this system actually represent what I need?
  • Cost and durability: what are the operational properties of this backend?
  • Collective wisdom: accumulated knowledge from millions of prior interactions baked into the model

An agent choosing between cloud platforms will use the same kind of reasoning a very experienced engineer would — actual technical properties, not how well the landing page explains the API. This means the path to being chosen by agents is building genuinely better systems, not better marketing to agents.

The 100–1000x Agent Multiplier

Current software was designed for human throughput. An enterprise of 5,000 employees maps roughly to 5,000 software users. In an agent-augmented world, each employee may run multiple agents concurrently — meaning the same enterprise now generates 50,000–5,000,000 software interactions, most of them automated.

This changes everything about system design:

  • Concurrency patterns that never mattered for human users become critical
  • Access control and identity need agent-native models (see below)
  • Monetization shifts toward usage-based pricing because token/call volume explodes
  • Rate limits, quotas, and audit trails take on entirely new importance

Box’s experience with their CLI: giving Claude Code access to a full organizational repository immediately surfaced coordination problems (concurrent writes, accidental moves, conflicting deletes) that human usage never triggered — because humans are slow.

The Agent-as-Extension-of-Self Problem

Agents are not employees. They don’t have privacy rights, they have no independent accountability, and — critically — they can be prompted or socially-engineered into revealing anything in their context window. The current operating model is that agents are extensions of the human who runs them, not independent actors.

Key implications:

  • You have full liability for what your agent does
  • You have complete oversight (and must be able to “log in as” your agent)
  • But: if you can log in as your agent, how can it operate securely in shared spaces? How can it receive confidential information from other agents?
  • Keeping something secret in a context window is an unsolved problem — “you should assume anything in the context window can be prompt-injected out of it”

This creates a fundamental tension: agents need enough context to be useful, but anything in that context is potentially leakable. Enterprise adoption will be throttled until this is resolved.

The analogy to open source: similar governance debates happened in conference rooms when open source emerged; the norms got developed. Those debates are now happening in public in real time.

Enterprise vs. Startup Diffusion Gap

The diffusion of AI capability will take longer than Silicon Valley assumes. The bottleneck is not technology — it’s organizational complexity and systems-of-record entrenchment.

Startups: can build from zero with no legacy, give agents full context, write software on the fly. Move fast.

Enterprises: face multiple compounding blockers:

  1. Existing systems aren’t going anywhere: SAP, Workday, legacy ERP. Domain knowledge is encoded not just in data but in UI, middle tiers, and usage patterns — you can’t vibe-code your way to decades of accumulated operational logic
  2. Integration risk: agents creating on-demand integrations between systems terrifies CIOs because it bypasses governance controls built up over years
  3. Context window security: confidential data in agent context = potential exfiltration vector
  4. Agent identity governance: who is responsible when agent A accidentally accesses data via agent B’s shared permissions?

The most exciting near-term scenario: first-principles knowledge-work companies (law firms, consulting, engineering shops, marketing agencies) built ground-up with agent-native architectures. No legacy systems to protect. Full context available to agents. These will become case studies for what the agent-first enterprise looks like — until they, too, accumulate legacy.

Algorithmic Thinking as the Human Bottleneck

Even with perfect software, most people cannot tell an agent what to do. In any team of 50 marketing people, probably one person can document their own workflow as a flowchart. Everyone else would fail — not because they’re bad at their job, but because their work is tacit knowledge that was never formalized.

The abstraction layer analogy: spreadsheets didn’t instantly replace rooms of manual calculators. There was a generation that had to become “spreadsheet people” before the abstraction became invisible. AI agents are at the same inflection — requiring a level of systems thinking that few people currently have, but which will become normalized over the next generation of workers.

Death of the UI

Andreessen’s long-run implication: if agents are the primary consumers of software, the entire rationale for UI design collapses. “Who is going to use software in the future? Other bots.”

The browser made sense because humans needed a readable, navigable interface. An agent has a shell — it doesn’t need a browser. The 402 Payment Required story follows the same logic: early internet missed payments because there was no frictionless way to transact; agents have zero payment friction, and stable coins provide internet-native money. “AI is the crypto killer app.”

In 10 years, the concept of a programming language may not be salient — models code in whatever is optimal, translate freely between languages, and may eventually emit binaries or model weights directly. “Human-built software systems compensate for human limitations. Remove those limitations and the abstractions change.”

Software Economics Inversion

Wall Street models for AI are wrong by at least an order of magnitude because they treat it as a fixed-size market shifting between providers. Historical pattern:

  • PCs: people thought it was a finite MIPS market; missed that software would be a separate industry
  • Cloud: people thought it was just server hardware in someone else’s data center; missed the 1000x consumption increase when marginal cost dropped
  • AI: people are modeling it as a rearrangement of existing spend; will miss the net-new value created when agent volume explodes

The compute budget question (“how much should we allocate to tokens?”) is the new cloud OpEx debate. It will feel urgent and unsolvable for 3–5 years, then become normalized just as cloud spend did.

Microtransactions unlock: agents, unlike humans, have zero friction for small payments. This is the first time you can put valuable information or computation behind a micropayment wall and have a buyer who will actually transact. New markets for data and computation that were economically unviable for human-facing products become viable for agent-facing ones.

High-Bandwidth Artifacts: What Agent-First UI Looks Like

The “death of the UI” (Andreessen) and the “100–1000x agent multiplier” (Levie) describe why current software design assumptions are wrong. But there’s a complementary question: what does the collaboration surface look like when agents and humans work together on complex tasks?

Jacob Lauritzen (Legora CTO) argues that chat is the wrong answer. Chat collapses an agent’s multi-branch work tree into a linear thread. For reviewing a 30-minute contract drafting run, chat offers no structural way to see what changed, where the agent made autonomous decisions, or how to surgically correct clause three without disrupting the rest.

The answer: high-bandwidth, persistent, domain-specific artifacts:

  • Legal: a document with tracked changes, inline commenting, per-clause agent tags, and handoff to specialized sub-agents
  • Legal review: a tabular review interface where the agent populates a structured comparison across contracts, flags items needing judgment, and the human scans + resolves
  • Coding: PR diffs with test results as the verification surface
  • Finance: financial models with variance annotations

These artifacts are not just better UX. They are the interface design response to the agent-first design principle: agents will be the primary producers; humans will be the reviewers and judgment-imposers. The review interface must match the domain’s natural review primitive — not the text-input primitive inherited from human chat.

Language-as-input is fine and very good (flexible, low-barrier). Language-as-the-only-collaboration-surface is a human limitation inappropriately imported into an agent-native workflow.

See agent-human-collaboration for the full trust/control framework.

Dark Code: The Risk Side

Sarah Guo names the failure mode of poorly governed agent-first software: dark code — behavior in production that nobody can explain end-to-end. When agents select tools at runtime and natural language is the control plane, execution paths don’t exist until they run and may never appear in source code.

The Levie “context window leakage” problem (anything in an agent’s context can be prompt-injected out of it) is one variant of dark code. Cross-tenant exposure via cached agent results is another. Both share the same root: agent-driven behavior that generates no clean audit trail and cannot be attributed to a single actor.

The accountability question that will arrive: “Can you say what your system did with a customer’s data on a specific Tuesday in March?”

See dark-code for the full framework.

Connections

  • model-context-protocol — MCP is one implementation of the agent software interface; agent-first software needs agent-native API design
  • agent-sandbox — per-user isolation is a partial answer to the agent-as-extension-of-self problem
  • s3-first-architecture — file-centric storage (like Box) is a natural fit for agent workflows; “every agent loves working with files”
  • agentic-engineering — the guardrails layer directly addresses the context-window leakage and permission problems discussed here
  • aaron-levie — Box CEO; the agent interface = human interface framing
  • andrej-karpathy — “apps shouldn’t exist; just APIs and agents” — Dobby home automation as demonstration; “the customer is no longer the human, it’s agents acting on behalf of humans”
  • marc-andreessen — “death of the browser/UI”; bots using software for bots; AI + crypto for agent payments
  • sarah-guo — dark code as the governance failure mode of agent-first software
  • dark-code — emergent, unattributable production behavior; the downside of the agent volume multiplier
  • agent-human-collaboration — trust/control framework; high-bandwidth artifacts as the collaboration surface in agent-first products
  • jacob-lauritzen — Legora CTO; high-bandwidth artifacts thesis (documents, tabular reviews > chat)
  • ai-agents — broader topic area

Sources

Graph View

Backlinks