Dark Code

Behavior in production that nobody can explain end-to-end — systems whose behavior emerges from runtime interactions between components, without anyone holding a complete mental model of what the system actually does.

Last updated: 2026-04-16

Overview

Sarah Guo’s term for a new class of systemic risk in AI-augmented production systems. Dark code isn’t just unreviewed code — it’s behavior that arises from agents selecting tools at runtime, natural language acting as a control plane, and execution paths that don’t exist until they run. It leaves no clean audit trail and cannot be attributed to a single actor.

The canonical example: a non-technical employee connects a customer data API into a reporting pipeline. An agent selects steps at runtime; one step caches results where another service can read them. Every individual service stayed within its permissions. Nothing was obviously misconfigured. The cross-tenant exposure only existed at runtime, assembled by an agent that no longer existed by the time anyone went looking. Four days to understand the incident.

This is structurally different from traditional bugs. It’s not an error — it’s an emergent property of a system too complex for its operators to hold in their heads.

What Makes It Dark

Three structural causes:

1. Runtime path assembly — agents select tools dynamically; execution paths don’t exist until they run, and may never appear in source code
2. Natural language as control plane — the thing determining behavior is a prompt interpreted in context, not deterministic logic
3. Distributed authorship — non-technical builders (marketing, PMs) can now create production-adjacent behavior; system-creating power is widely distributed, but accountability and review are not

The builder population has expanded enormously. The risk surface has expanded with it, without the corresponding review infrastructure.

The Attribution Problem

When something goes wrong with dark code:

• Who did this? — A workflow someone set up + an agent executing it + a chain of tools. You can reconstruct what happened from logs. You can’t cleanly assign it to a single actor.
• What did the system actually do? — Current SOC 2 / security tooling was designed for systems whose complexity was at least deliberate. Dark code generates novel behavior as part of normal operation.
• Prompt injection as exfiltration — content embedded in external data (a web form, a document) can direct an agent to exfiltrate data through a trusted domain. Salesforce Agentforce had this vulnerability.

Guo’s framing: “Charles Perrow had a term for failures like these: ‘normal accidents.’ Not caused by error or negligence, but built into the structure of any system too complex for its operators to hold in their heads.”

The Accountability Gap

The questions procurement currently asks: SOC 2 compliance, encryption at rest — controls designed for a world where you could enumerate what software does.

The questions they should ask (and will, eventually):

• What are your agents doing in production?
• Can you reconstruct a decision path?
• Can you say what your system actually did on a specific Tuesday in March?

The SOC 2 report looks the same whether a system has 3 agent-driven workflows or 300. That’s not good enough.

Why It’s Getting Worse

• Speed — code is being produced faster than understanding can catch up; tests pass, diffs look clean, but nobody ever understood the system as a whole
• Shadow operators — non-engineers wiring tools directly to production data through increasingly accessible interfaces
• Security layer exposure — dark code in applications is a liability; dark code in secrets management, identity, and access control is an existential risk

The comparison to SaaS sprawl/shadow IT undersells it: previously, people were connecting existing services. Now they’re creating new behavior, and that behavior can come into existence when someone describes it in English.

What Good Looks Like

Guo doesn’t prescribe a complete solution, but the implied requirements:

• Structured agent logging — not just tool calls, but decision paths and context that explain why a path was taken
• Runtime audit trails — replay capability for any agent-driven execution, not just log tailing
• Agent identity governance — clear model of who is accountable for what agents do, separate from the human who set them up
• Scope constraints at the harness level — see agentic-engineering Guardrails layer; agent-sandbox

The key question isn’t whether you had good intentions. It’s whether you maintained the ability to explain what your system actually did.

Connections

• agent-first-software — the same agent volume explosion that creates business opportunity also creates dark code surface area; Levie’s “context window leakage” problem is dark code by another name
• agentic-engineering — Guardrails layer (Layer 4) is the direct engineering response to dark code risk
• agent-sandbox — per-user isolated execution environments reduce cross-tenant blast radius
• sarah-guo — Conviction VC; coined the term; portfolio-wide pattern recognition
• ai-agents — dark code is the risk side of the agentic era upside

Sources

• Dark Code — Sarah Guo, Conviction — added 2026-04-16