Context Engineering: Why Hayek's Knowledge Problem Survives AI

AI can process anything. The hard part is deciding what it needs to know.

Will the AI economy be a centralized one? Not in a dystopian science fiction sense, but in the practical sense that matters to most people: will a handful of organizations, wielding intelligence that can process more information than any human workforce, make the decisions that shape our industries, our firms, and our daily work? Will the efficient path be to concentrate decision-making in fewer hands, because that is where the processing power lives? This is the question that sits beneath the surface of the most important recent academic treatment of AI and centralization: Erik Brynjolfsson and Zoë Hitzig’s “AI’s Use of Knowledge in Society,” a chapter in the forthcoming University of Chicago Press volume The Economics of Transformative AI.

I have been thinking about this chapter since it was published last September, and I keep coming back to a tension between its theoretical framework and what I observe in practice. Brynjolfsson, a Stanford economist and leading scholar of information technology’s effects on productivity and organizations, and Hitzig, a researcher at Harvard’s Society of Fellows, make a rigorous case that AI undermines the classic argument for economic decentralization. Their logic is persuasive at the level of their framework. But something is missing from the framework, and it is something that the people building enterprise AI systems encounter in practice: the cost, effort, and human judgment required to make knowledge usable by AI and to keep it usable as domains evolve.

That missing something has a name. In September 2025, the same month as their chapter, Anthropic published a technical primer on what it called “context engineering.” The term describes something more involved than prompt engineering, which is about writing effective instructions for a model. Context engineering is the work of selecting, structuring, and continuously updating the information that reaches an AI model when it performs a task, including data from external systems, prior conversation history, tool outputs, and specialized knowledge about the field the model is operating in.

The framing of the Anthropic primer is what caught my attention. Context, they write, “must be treated as a finite resource with diminishing marginal returns.” LLMs have an “attention budget” that depletes with every additional token. Good context engineering means finding “the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.” And: “opinionated and thoughtful engineering is required to ensure that an LLM has the right tools and heuristics for effectively navigating its information landscape. Without proper guidance, an agent can waste context by misusing tools, chasing dead-ends, or failing to identify key information.”

This is the language of economic tradeoffs. The people building the most capable AI in the world are telling us that making knowledge usable by AI is itself a scarce and costly activity requiring human judgment.

This essay argues that the economics of context engineering expose a gap in the Brynjolfsson-Hitzig framework that changes its practical implications: for how enterprises build with AI, which firms centralize successfully, and whether the AI economy will be as centralized as their framework suggests.

The stakes are not abstract. On February 24, Fed Governor Lisa Cook told the National Association for Business Economics that “we appear to be approaching the most significant reorganization of work in generations,” warning that job displacement may precede job creation in ways monetary policy cannot fix. The same day, Anthropic co-founder Jack Clark told Ezra Klein that “the value of more senior people with really well-calibrated intuitions and taste is going up, and the value of more junior people is a bit more dubious.” The question of who holds valuable knowledge, and how that knowledge gets centralized for AI processing, is no longer just a question for economists. It is a question for anyone whose work involves judgment.

Hayek’s Problem, Revisited for AI

Start with the argument Brynjolfsson and Hitzig are updating. In 1945, the Austrian economist Friedrich Hayek made an epistemic argument against central planning. The problem with centralized economic coordination, Hayek argued, is that the knowledge required for efficient economic decisions is inherently dispersed across society. Much of this knowledge is local (”knowledge of the particular circumstances of time and place”) and tacit (it resists being written down or transmitted). The experienced portfolio manager who senses when a deal is overpriced, the senior lawyer who intuits which clause a counterparty will resist, the factory foreman who knows which machine is about to fail: this knowledge lives in people, embedded in their practice. It cannot simply be reported upward for centralized decisions about capital allocation, production, or strategy. Hayek’s conclusion: because you cannot centralize the knowledge, you should not centralize the decisions. Decentralized decision-making, where local actors use local knowledge, produces better outcomes than centralized coordination by any single entity.

Brynjolfsson and Hitzig argue that AI fundamentally changes the terms of Hayek’s problem through two channels.

First, AI expands the frontier of what knowledge can be codified. Knowledge that was previously tacit and locked in local decision-makers’ heads (what they call “inalienable” knowledge, borrowing from property rights theory) can increasingly be captured, digitized, and transmitted to headquarters (”alienable” knowledge). Their framework concerns how firms organize decision-making between a central headquarters and local operating units (think: a retail chain and its stores, an investment firm and its portfolio managers, a law firm and its practice groups). When the local manager’s knowledge is inalienable, efficiency favors giving that manager autonomy. The experienced portfolio manager should make her own allocation decisions, because only she has the accumulated judgment about her market that no report can convey. But when AI makes that knowledge capturable and transferable, headquarters can absorb it and make decisions centrally across the firm, and the experienced portfolio manager may no longer be necessary.

Second, AI massively expands the information processing capacity of the center. In their framework, this is captured by the parameter K̄, which measures how many information flows (say, daily sales reports from each store, or real-time position data from each portfolio manager) headquarters can aggregate, interpret, and act on. As K̄ increases, the case for centralization strengthens: headquarters can realize coordination gains, the value created when decisions across different units are made in concert rather than independently, across more and more of the firm’s assets.

Their predictions follow: if the center can absorb more knowledge and process more information, then a single headquarters can effectively coordinate more assets, more divisions, more decisions. This means larger firms (measured by what a single organization can control, not necessarily headcount), greater industry concentration, and reduced local managerial autonomy. In short: a more centralized economy, where coordination is increasingly handled by AI. They review empirical evidence, much of it from the pre-LLM era, and find it broadly consistent: rising market concentration, the spread of centrally controlled franchise models, IT-intensive firms growing larger.

The Lossless Assumption

Brynjolfsson and Hitzig recognize that codification requires real infrastructure investment, and they acknowledge that some knowledge may resist codification indefinitely. But their framework does not give the codification process itself an economic structure.

So what, precisely, is missing? The framework makes three interrelated assumptions about codification: that it is binary (knowledge is either inalienable or alienable, with no intermediate state), lossless (once alienable, the full coordination gains follow as if the informational barrier had never existed), and costless (codification requires no significant upfront investment in quality and no ongoing maintenance after the transition). These assumptions reinforce each other. If codification is binary, there is no such thing as partially codified knowledge: knowledge that has been captured but has lost some of its value in the transition. If there is no quality to degrade, there is no cost to producing quality, and no maintenance cost. The question is whether any of these assumptions hold in practice.

Here is how it works in their framework. Knowledge is either inalienable (locked in the local manager’s head, favoring decentralization) or alienable (captured and transferable to headquarters, favoring centralization). Once knowledge crosses that threshold, the framework treats it as fully usable by the center: the coordination gains are the same as if the informational barrier had never existed.*

What the framework does not formalize is the cost and quality of making knowledge usable to the center. Not just capturing raw information (deploying sensors or recording Zoom calls), but the harder work: investing in codification quality upfront, maintaining it as domains evolve, and curating the right codified knowledge for each task. The framework acknowledges the space between “not readily codifiable” and “codified and ready for centralized processing,” but it does not give that space its own economics.

Context engineering is the emerging discipline that lives in that space. And its economics are not trivial.

The Economics of Context

The Anthropic primer, introduced earlier, frames context as an optimization problem under constraint. The constraint is finite model attention, and the objective is maximizing the return on the context you invest. The return depends not on how much information you feed the model but on how well that information is selected, structured, and kept current for the task at hand. The Anthropic primer describes one consequence of getting this wrong: “context rot,” where the model’s ability to use information degrades as the volume of context grows. But the deeper economic point is that context quality is a variable, not a constant, and improving it requires investment that the Brynjolfsson-Hitzig framework does not formalize.

To see where this investment enters the framework, return to the two channels through which Brynjolfsson and Hitzig argue AI tips the balance toward centralization.

The Missing Term

Call the cost of this investment cCE: the cost of context engineering. It enters both channels.

Channel 1: codification. The framework treats codification as lossless: once knowledge is alienable, the full coordination gains follow. But codification quality is not binary; it is a continuum, and where you land on that continuum depends on how much you invest. That investment is cCE in its first form: the cost of codifying knowledge well rather than merely capturing it.

The experienced lawyer’s judgment about which provisions matter does not arrive at headquarters fully intact just because her calls have been recorded and her redlines tracked. Much of what makes her judgment valuable is never captured in any artifact: the hesitation she noticed in a client’s voice when discussing indemnification, the instinct that a counterparty’s outside counsel is posturing on a liability cap rather than drawing a real line.

The natural response is to capture everything: record every call, track every redline, log every email. But total capture can be worse than incomplete capture. Consider a legal team at a fast-growing startup that spent its first three years agreeing to risky contract positions — aggressive indemnification terms, loose liability caps — in order to close deals and grow the business. Those contracts are now in the system. If you prompt a model on the full history, you are teaching it that those positions are normal. It will reproduce the past’s expedient decisions as if they were the company’s standards. Someone with domain knowledge must decide what the model should learn from and what it should ignore, and that judgment is not in the data. It is in the head of the lawyer who lived through the transition. The cost of accessing that judgment is a concrete component of cCE.

This reveals a dependency the framework does not account for: the knowledge required to make codification useful rather than merely complete is itself inalienable, residing in people who understand the domain's history and evolution. It is why codification has a cost that does not go to zero.

Channel 2: processing capacity. Their framework’s K̄ parameter counts how many information flows headquarters can process, and each processed flow delivers the full coordination gain. K̄ is a quantity measure. It does not quality-adjust the flows.

But even if every piece of local knowledge were perfectly codified, a second problem remains: which codified knowledge should the model see for a given task? A model reviewing contracts needs different context depending on the type of contract, one’s role in the transaction (e.g., buyer vs. seller), and the provisions being negotiated. The challenge here is not codification but curation: selecting and assembling the right bundle of already-codified information at inference time. This is cCE in its second form. It requires ongoing judgment about what the model needs to see for this question, in this domain, right now. Increasing K̄ lets headquarters process more, but the value of that processing depends on what is fed to it.

The link between the channels. This is the key point: the two channels Brynjolfsson and Hitzig identify are not independent. They are linked by context quality. Expanding K̄ only delivers its full centralization benefit if the codified information being processed is both high quality and well-curated for the task. Channel 2 (processing capacity) is gated by the quality of Channel 1’s output (codification), and both are gated by context engineering investment that requires local domain expertise.

cCE spans codification and central processing. It encompasses both the cost of converting tacit knowledge into usable form (with inevitable quality loss) and the cost of curating and assembling the optimal bundle of codified information at inference time. It mediates the transition from inalienable to alienable knowledge, it requires local domain knowledge from humans, and it does not go to zero as K̄ increases.

What this means for the framework. If cCE is low and declining, the Brynjolfsson-Hitzig predictions hold with a lag: centralization proceeds as models improve, just with some transitional friction. But if cCE is durable, if it regenerates at each new level of AI capability because each new capability opens new use cases that require their own context engineering, then the framework’s two channels are permanently attenuated. The coordination gains from centralization are real but smaller than the framework predicts, because they are multiplied by a quality factor that depends on costly, ongoing, expertise-dependent investment the center cannot simply command. The result is not that centralization fails, but that it succeeds unevenly: fully for the routine and easily codifiable, partially and expensively for everything else.

As models become more capable, organizations deploy them on harder, more judgment-intensive tasks, and each new class of task requires its own context engineering. Capability without relevant context produces confident but misaligned output. And the investment is not one-time: even within existing tasks, the systems that produce good context depreciate as domains evolve, competitive landscapes shift, and strategic priorities change. Maintaining context quality requires continuous input of local knowledge.

The Strongest Objection

Context engineering is not the only path from inalienable to alienable. Fine-tuning and reinforcement learning offer a different approach: adapting a model’s weights to a specific domain rather than curating information at inference time. Companies like Mercor, valued at $10 billion, are building infrastructure for domain experts to train AI models through evaluation and feedback, baking expert judgment directly into the model. If fine-tuning and RL could fully capture domain expertise, the bottleneck I describe would be transitional, because the experts’ knowledge would literally become part of the model. (Though even here, the same RL-trained models available to large headquarters are also available to smaller firms with deep domain expertise.)

But fine-tuned and RL-trained models share a limitation: their domain knowledge is frozen at the moment of training. Local knowledge changes fast. The emerging consensus among practitioners is: use fine-tuning and RL to build the foundation (what you might call the “trunks” of domain knowledge), and use context engineering to handle the “twigs,” the local, contextual, rapidly changing specifics. (I borrow the trunks/twigs metaphor from Hollis Robbins and her Substack essay “AI and the Last Mile.”) The trunks centralize. The twigs require someone with local, domain-specific knowledge.

I take this objection seriously. If Mercor-style expert training can capture not just stable professional knowledge but the rapidly evolving, firm-specific judgment that context engineering currently requires, then I am wrong. I do not think the current evidence supports that conclusion.

What the Work Actually Looks Like

Whether the context engineering bottleneck is durable is ultimately an empirical question. Here is what the work looks like in practice.

Context engineering as a discipline is new because powerful LLMs are new: the specific challenge of curating information for a model that reasons over a finite context window did not exist before transformers. But the underlying economic activity, translating domain expertise into structured data that a system can act on, has precedents. At Palantir, I was embedded with Fortune 100 companies doing a version of this for analytics platforms. Working in legal AI now, I do it for LLMs. The intellectual work in both cases lives upstream of the technology.

One concrete example. I built an internal strategic analysis pipeline: a system that synthesizes what prospective customers actually say across hundreds of sales conversations, combined with product usage data, to inform company strategy. The raw inputs, call transcriptions joined with CRM metadata, are Channel 1: inalienable knowledge (what individual salespeople heard) codified into data. But at this stage the codification is not yet usable, too large and too noisy to produce usable output.

Channel 2 is where the work gets hard. I built a processing pipeline to parse, filter, and structure transcripts before they reached the LLM, then a system prompt encoding a compressed theory of what matters strategically. That prompt required deep familiarity with the product, the market, and leadership’s strategic questions. No foundation model could have set those priorities on its own. They were particular to our company, our competitive position, and our strategic moment.

The analysis surfaced something no one on the team had seen: prospects in a specific segment were independently describing the same unmet need in almost identical language, but our messaging was organized around a different value proposition entirely. The pattern was invisible at the level of any individual call. It only emerged through a pipeline designed to surface patterns the team hadn’t thought to look for. That finding changed our product prioritization and go-to-market strategy. The AI did the processing. The context engineering is what made the processing valuable.

I believe this pattern generalizes. The two channels from the Brynjolfsson-Hitzig framework appear in practice as two distinct gaps where human judgment creates the most value. The first is problem formulation: defining the strategic question and specifying what data should exist to answer it. This is the Channel 1 gap: deciding what to codify and how to codify it well requires understanding the business deeply enough to know which questions are worth asking. The second is curation: assembling the right codified information for a specific AI task. This is the Channel 2 gap. It requires domain judgment about what the model needs to see and what it should ignore. In an economy where highly capable AI is a commodity, these two gaps are where knowledge work lives.

What Context Engineering Means in Practice

The claim that context engineering is a durable bottleneck with specific economics has implications for several live questions about AI and economic organization.

The future of SaaS. When Anthropic launched Claude Legal last month and $285 billion in SaaS market cap evaporated in a day, the market thesis was: general-purpose AI replaces domain-specific software. Context engineering complicates this. A general-purpose model, even one improved by reinforcement learning from domain experts, still requires domain-specific context at inference time to be useful for a particular firm’s particular questions. The SaaS companies that survive may be those that have accumulated, over years, the domain knowledge, data relationships, and customer proximity needed to build high-quality context in their verticals. Code was never the moat; context was.

But the disruption is also real: companies that are not in the software business may increasingly build their own context engineering capabilities internally. An auto manufacturer that deeply understands its quality assurance processes may be able to construct its own AI pipeline for defect analysis, without purchasing vertical software, especially as AI-assisted coding lowers the barrier to building data pipelines.

Recent data from Menlo Ventures is instructive here. Enterprises spent $19 billion on AI applications in 2025, with 76% of AI use cases purchased rather than built internally, up from 53% a year earlier. This suggests that, despite the declining cost of building, there is real value in the application layer that sits on top of foundation models, and at least some of that value is context engineering: the domain-specific context pipelines, task-level integrations, and curation logic that make a general model useful for a particular job.

What is definitive is that context engineering capability, wherever it lives, is becoming a primary source of competitive advantage. The firms that thrive will be those that combine domain knowledge with the ability to translate that knowledge into high-quality AI context, whether they sell that capability as a product or deploy it internally.

Forward deployed engineers. In an earlier essay, I noted that forward deployed engineer job listings grew more than 800 percent between January and September 2025, according to the Financial Times. Palantir pioneered this approach: its business model depends on embedding engineers (which it calls “forward deployed engineers”) with customers to learn their domains and structure their data into usable representations. Context engineering provides the economic logic for this pattern: forward deployed engineers are context engineers for their customers’ domains. They learn which data flows matter, which workflows are real versus nominal, and which problems are worth solving.

Knowledge workers. The Brynjolfsson-Hitzig framework, updated with cCE, suggests that the AI disruption of knowledge work will be significant but uneven. Foundation models and domain-specific models trained through reinforcement learning are automating the “trunks” of knowledge work: the routine, easily codifiable parts of legal review, financial analysis, and medical diagnosis. But the “twigs,” the local, contextual, rapidly evolving questions that drive strategic value, require humans who understand the domain well enough to engineer effective context. This creates a new kind of knowledge worker: one who has done enough of the underlying work to know what good looks like, and whose value lies in framing the right problems, prioritizing what data needs to be captured, curating and assembling it for AI consumption, and evaluating whether the AI output actually makes sense.

People preparing for this economy should invest in domain expertise of the twig variety: not memorizing standard contract templates (those are trunks, and they are being automated), but developing the judgment to recognize, say, that a particular counterparty’s negotiating pattern has shifted in a way that changes which provisions matter this quarter. They should also invest in practical data skills: basic data storage and database concepts, pipeline construction, data transformation and integration. The person who can both understand a domain deeply and package that understanding for AI consumption will have the most durable role.

The harder question is how entry-level workers develop twig expertise when the trunk work that traditionally served as training ground is increasingly automated. This is an institutional challenge, not just an individual one.

One approach: structure entry-level roles around context engineering itself. Have junior people work alongside senior domain experts to build and maintain the data pipelines, evaluation rubrics, and context configurations that feed AI systems. This is less radical than it sounds. Knowledge work has always been about synthesizing large amounts of information and turning that synthesis into action: executing a trade, making a redline, advising a client. The difference now is that the synthesis must be legible not just to other humans but to models.

Structuring entry-level work around that translation is the modern equivalent of apprenticeship. The junior person learns the domain not by doing the routine work that AI now handles, but by learning to judge AI output, identify where context is missing or wrong, and iterate on the systems that produce it. The trunk work becomes the AI’s job; learning to tend the twigs becomes the human’s training ground.

Conclusion

This returns us to the question that opened this essay: will the AI economy be a centralized one?

The Brynjolfsson-Hitzig framework says the forces point toward yes. But Avi Goldfarb, a University of Toronto economist and one of the leading scholars of AI’s economic effects, observed in his discussant comment on the chapter that the same AI which centralizes headquarters’ processing power also distributes the tools of centralization to smaller actors.

Context engineering provides a mechanism for why Goldfarb’s counterpoint has teeth: if the binding constraint is not processing power but the quality of what gets processed, then smaller organizations with deep domain knowledge and strong customer relationships can compete by building better context, even against larger firms with more powerful models. Context engineering is not like building a data center, where scale economies favor the largest player. It is like learning a client’s business, where proximity and relationship depth matter more than budget.

So will the AI economy be centralized? Partially, and increasingly, for the trunks. But at the twigs, where context engineering determines which firms create value from codified knowledge, there is room for a more distributed economy than the framework alone predicts. The answer depends less on the power of the models and more on which kinds of firms invest in the human and organizational infrastructure to feed them well. Hayek’s insight survives in a different form: local knowledge still matters, but now it matters as an input to context engineering rather than as an input to local decision-making. The knowledge problem has not been solved. It has migrated upstream.

Predictions

If context engineering is a durable bottleneck rather than a transitional friction, several things should follow.

1. By 2028, the enterprises seeing highest ROI from AI will be those that invested in context engineering capabilities (domain specialists, data curation systems, customer embedding practices) before or during AI deployment. Correlation between context engineering maturity and AI ROI will exceed correlation between AI spending and AI ROI.

2. By 2028, at least two major enterprise software categories will emerge specifically around context engineering, distinct from both data infrastructure and AI model providers. One will focus on context curation and compression: tools that help organizations select, structure, and maintain the domain knowledge that feeds AI systems. Contextual AI, whose CEO told VentureBeat in January that “the bottleneck is context,” is building exactly this. The other will focus on context evaluation: tools that measure whether AI outputs are actually good, which requires embedding domain-specific quality standards into automated assessment. Braintrust, which just raised $80 million at an $800 million valuation, and Arize AI, with $131 million in total funding, are early leaders here. The “context layer” becomes a recognized market segment. These companies exist today, but they are not yet recognized as a coherent category in the way that “data infrastructure” or “AI model provider” are. My prediction is that they will be by 2028.

3. The SaaS companies that outperform through 2028 will be those with the deepest domain-specific context engineering capabilities, measured by customer proximity and domain data assets, not those with the most features or broadest integrations.

4. By 2029, average firm size in AI-exposed industries will not increase uniformly as the Brynjolfsson-Hitzig framework predicts. Industries will concentrate at the infrastructure layer (cloud computing, model training) but fragment at the application layer, as smaller firms with superior context engineering in specific domains outperform larger organizations with generic context.

5. The share of AI-adjacent roles focused on data curation, domain-specific AI configuration, and context engineering will grow as a proportion of total knowledge work through 2028, even as total knowledge work headcount in some functions declines.

If these prove wrong, and if general-purpose AI with RL-trained domain expertise can handle complex, context-dependent enterprise workflows without significant ongoing human context engineering by 2028, then the Brynjolfsson-Hitzig centralization thesis is stronger than I think, and the knowledge economy reorganizes more dramatically than the context engineering bottleneck would suggest.

If they prove right, then the framework needs its additional term: cCE, a cost of context engineering that attenuates the coordination gains from both codification and expanded processing capacity, that requires local expertise to produce, and that creates a durable role for humans with domain knowledge in the AI economy.

Context engineering is not a guarantee that knowledge workers will be fine. If fine-tuning and RL advance faster than I expect, the bottleneck I describe may prove transitional. But even the people building the most capable AI systems see something like this bottleneck in their own organizations. Jack Clark’s observation to Ezra Klein last week that senior people with “well-calibrated intuitions and taste” are gaining value at Anthropic is a description of context engineering in practice: experienced judgment about what matters, applied to an environment where AI does the processing.

AI is already transforming knowledge work. The question is whether that transformation will be mediated by a durable bottleneck with specific economics, or will be a fast transition to a world where general-purpose AI handles context on its own. I believe the bottleneck is real, and that understanding its economics is essential for firms, workers, and policymakers making decisions now.

*Footnote: Formally, Section 4.3 of their chapter shows that once local information assets are codifiable and transferable, “the coalition values with bundles are the same functions” as in the pure coordination case, and “the relevant first-order conditions revert exactly” to the equations that favor centralization.