How to Build a Context Graph

Introduction

The answer to building context graphs isn't “add memory to your agent” or wire up MCP. In fact, the word “graph” itself is a little misleading. What you're really trying to model is far more dynamic and probabilistic than a static graph suggests.

Context graphs don't exist today because building them forces us to confront problems we've spent decades ignoring. Every organization pays a fragmentation tax: the cost of manually stitching together context that was never captured in the first place.

Different functions use different tools, each with its own partial view of the same underlying reality. A context graph is infrastructure to stop paying that tax. But to build one, you first have to understand why the tax exists.

The Two Clocks Problem

There's an intuition that helps explain why building context graphs is so difficult: we've built all our systems around only half of time.

“The config file says timeout=30s. It used to say timeout=5s. Someone tripled it. Why? The git blame shows who. The reasoning is gone.”

This pattern is everywhere:

• ●The CRM says “closed lost.” Doesn't say you were the second choice and the winner had one feature you're shipping next quarter.
• ●The treatment plan says “switched to Drug B.” Doesn't say Drug A was working but insurance stopped covering it.
• ●The contract says 60-day termination clause. Doesn't say the client pushed for 30 and you traded it for the liability cap.

Why This Is Hard

Agents As Informed Walkers

The ontology problem looks unsolvable at first. Every organization is different. Every system has unique structure. You can't standardize “how decisions work” any more than you can standardize “how companies work.”

But there's something that navigates arbitrary systems by definition: agents.

When an agent works through a problem (investigating an issue, making a decision, completing a task), it figures out the relevant ontology on the fly:

• • Which entities matter?
• • How do they relate?
• • What information do I need?
• • What actions are available?

There's an intuition from graph representation learning that's helpful here. Graph embeddings (like node2vec) showed you don't need to know graph structure to learn representations of it. Random walks—sequences of nodes visited by wandering through edges—are sufficient.

Agents are informed (not random) walkers. Unlike random walks, agent trajectories are problem-directed. The agent adapts based on what it finds. Investigating a production incident, it might start broad, then narrow as evidence accumulates. Random walks discover structure through brute-force coverage. Informed walks discover structure through problem-directed coverage.

Context Graphs as Organizational World Models

A world model is a learned, compressed representation of how an environment works. It encodes dynamics (what happens when you take actions), captures structure (what entities exist and how they relate), and enables prediction (given current state and a proposed action, what happens next?).

A context graph with enough accumulated structure becomes a world model for organizational physics. It encodes how decisions unfold, how state changes propagate, how entities interact. Once you have that, you can simulate.

“Simulation is the test of understanding. If your context graph can't answer ‘what if,’ it's just a search index.”

This is what experienced employees have that new hires don't. Not different cognitive architecture—a better world model. They've seen enough situations to simulate outcomes. “If we push this Friday, on-call will have a bad weekend.” That's not retrieval. It's inference over an internal model of system behavior.

Three Problems to Solve

Building a context graph requires solving three fundamental problems:

Practical Steps

The companies that succeed in building context infrastructure will have something qualitatively different: not agents that complete tasks, but organizational intelligence that compounds. That simulates futures, not just retrieves pasts. That reasons from learned world models rather than starting from scratch.

1. 1

   Start Capturing Decision Events

   Instrument the points where decisions are made. Not just the outcomes, but the context, alternatives considered, and reasoning applied.

2. 2

   Build the Event Clock

   Create infrastructure for temporal validity. Facts with validAt and invalidAt timestamps. The ability to query historical state.

3. 3

   Deploy Agents and Capture Trajectories

   Let agents solve real problems. Capture their traversal patterns. Learn structure from how work actually gets done.

4. 4

   Build Toward Simulation

   As patterns accumulate, test whether you can answer “what if” questions. That's when you know you've built something real.