The Copilot Fallacy

The Fallacy

Most institutions I advise are running a first-order strategy for a third-order problem.

Someone signs a Copilot or ChatGPT Enterprise contract. People start using it. Email drafts appear faster, slide decks assemble themselves, research notes take less of the day. Efficiency dashboards tick up. The transformation is declared under way.

This is the Copilot Fallacy: the systematic mistake of treating AI as software to license rather than as intelligence that demands reorganising the work around it. The name is not a critique of any specific product. Swap Copilot for Claude, Gemini, or an in-house model and the fallacy is unchanged, because it lives in the posture, not the tool. The reflex is to buy, deploy, measure per-seat productivity, and stop there.

That reflex is not irrational. Per-seat productivity is what IT procurement is built to defend, and vendor marketing materials are engineered to fit exactly that frame. But it is only the first of three orders of effect. The second and third orders are where competitive outcomes are actually decided, and they are where almost no institution I see is looking.

The rest of this essay is about those three orders: what each one is, why stopping at the first one is a strategic error, and what it takes to run all three.

How I Arrived at This View

The argument above is my own, but it is not armchair speculation. It rests on two pieces of work that I ran in parallel through late 2025 and early 2026: a systematic literature review of AI adoption in financial institutions, and a field study that I report only at the aggregate, methodological level here. The purpose of this section is to make the empirical scaffolding visible, not to relitigate the findings.

Systematic Literature Review (PRISMA 2020)

A PRISMA 2020 protocol guided the review across four databases (Semantic Scholar, OpenAlex, CrossRef, Google Scholar) and six specialist journals in financial innovation, digital banking, and electronic markets, covering the period 2019 through 2026. The dual-pathway design was deliberate: a database-only search systematically underrepresents practitioner-oriented contributions that show up in specialist venues the major indexes do not cover well.

Field Study and Analytical Structure

In parallel with the literature work I ran a field study with professionals across equity research, corporate finance, sales and trading, buy-side investment management, and adjacent functions, between late 2025 and early 2026. The specific content of those conversations stays confidential: nothing in this essay is attributable to any individual or institution. What I do surface here is the analytical structure that emerged from coding the material using a standard Gioia approach.

That structure produced five aggregate dimensions. Three of them map cleanly onto the three orders of effect and become the three pillars discussed later in this essay. Two emerged inductively and function as boundary conditions on the transformation sequence rather than additional pillars.

The two boundary conditions matter because they set the bounds inside which the three orders have to be navigated. Institutional embedding (regulation, legacy infrastructure, approval topology) sets the pace. Relational infrastructure (client trust, tacit market feel) sets a ceiling on what can be encoded and a direction for where freed capacity is most productively redeployed. Neither is an additional pillar. Both condition how the three-pillar sequence has to be adapted to each institution's situation.

1stFirst-Order Effects: The Productivity Lift

First-order effects are the ones you can put on a quarterly slide. An analyst drafts a note materially faster. A banker responds to mail in a fraction of the previous time. A developer accepts a meaningful share of Copilot suggestions. Every vendor in the AI stack publishes studies claiming some version of these numbers, and many of them are directionally right.

The uncomfortable fact is that first-order effects are not strategic. They are individual and additive. In a market where access to foundation models is priced by the API call and available to anyone with a credit card, a faster email is not a moat. It is a capability your competitor can license on the same day as you, by calling the same vendor, at the same price. The lift is real. The differentiation is not.

Worse, first-order metrics distract from the structural question. When leadership looks at the Copilot dashboard and sees adoption curves rising, the inference is: the programme is working. But the programme is only generating value of a kind every competitor can replicate by signing the same contract. The activity is real. The position is not.

The Copilot Fallacy, in its purest form, is the decision to optimise at the first order of effect only, treating the second and third orders as someone else's problem or a later phase that never comes. Once the procurement is in place and the dashboards are wired up, the institution has proven to itself that it is "doing AI," and the organisational energy for the harder work evaporates.

2ndSecond-Order Effects: Role Restructuring and the Capacity Flip

Second-order effects appear when enough people in a role use AI that the role itself starts to deform.

In knowledge work, the deformation has a direction. The routine, labour-intensive portion of a job, the data aggregation, the first-draft writing, the format-and-file cycle, collapses toward zero marginal time. What remains is what foundation models cannot reliably do without a human in the loop: interpretive judgment, contextual calibration, the recognition of when an output is wrong in a way that matters. Value redistributes from the ability to produce output toward the ability to curate and shape it.

I call this the Capacity Flip: the structural inversion of time allocation between routine and strategic work once AI is properly embedded. Before the flip, a financial analyst might spend roughly three quarters of the working day on routine production, a quarter on interpretive work. After the flip, the ratio inverts. The job title is unchanged. The job is not.

The flip does not happen automatically. It happens when the institution redesigns the workflow to capture the freed capacity and redirect it into the higher-value activity. Without that deliberate redesign, freed capacity gets eaten by meeting creep, reporting overhead, or simply more of the same routine work done in parallel. The productivity dashboard still rises. The job is still the old job. Nothing second-order has happened.

The Copilot Fallacy, at the second order, is the failure to notice that the point of AI-driven efficiency is to rebalance what the role is for, not to produce more of what the role used to do. Second-order transformation requires naming the flip, committing to it, and making it the basis for how the role is hired, measured, and promoted. That is an organisational intervention, not a technological one, and in my experience it is the point at which most AI programmes stall.

3rdThird-Order Effects: Architectural Parity and Encoded Judgment

Third-order effects appear at the market level, once second-order restructuring has diffused across enough institutions.

The decisive observation at this order is what I call architectural parity. Three, four, maybe five foundation models sit at the frontier at any given time, and every serious institution has access to all of them. The marginal capability difference between the top models is real but narrow, and it is shrinking. Model choice, in strategic terms, is commoditising.

If the model is not where advantage lives, then advantage has to live in what is encoded around the model: proprietary valuation frameworks, sector-specific analytical methods, institutional quality standards, accumulated judgment about which questions to ask, which inputs to trust, and which shortcuts are unacceptable. All of this exists in any serious institution. In most of them, it exists tacitly: in the heads of senior people, in unwritten stylistic rules, in templates no one has formalised, in conventions every junior professional absorbs by osmosis and none of them could articulate.

Third-order transformation is the systematic conversion of that tacit institutional knowledge into structured, AI-readable form, hosted on infrastructure the institution controls. I call this encoded judgment. Two things follow from it.

First, encoded judgment is what differentiates institutional AI output from commodity AI output. A vendor's generic model, pointed at a generic prompt, cannot replicate a methodology the institution itself has never written down. Once the methodology is encoded, the output carries the institution's signature in a way no external system can reproduce, and the case for human sign-off becomes defensible rather than ceremonial.

Second, encoded judgment compounds. Every additional piece of institutional expertise that gets structured into the system deepens its reach. Every application of the system surfaces cases where the encoding is incomplete, driving the next cycle of encoding. A procured generic tool produces no such compounding. It depreciates at the pace of the underlying model and reverts to commodity capability when the vendor releases the same integration to everyone else.

Sovereignty is the architectural complement. If the institution's differentiating knowledge is encoded into vendor infrastructure it does not control, the differentiation is rented, not owned, and it is rented from a counterparty with a commercial interest in keeping the tenant dependent. The Copilot Fallacy, at the third order, is the assumption that procurement decisions at the vendor level do not foreclose architectural decisions at the institutional level. They do. Every prompt that embeds proprietary methodology into a third-party API is a small forfeiture of sovereignty, and in aggregate these forfeitures amount to giving the vendor the thing that was supposed to distinguish you.

Sovereign AI Mastery: The Synthesis

The three orders of effect map onto three organisational interventions, in sequence. Taken together, they are what I call the Three-Pillar Model.

The sequence reflects a dependency structure, and the sequencing is not negotiable. Education is the work of teaching a role what AI actually does to it, and it is what makes the second-order shift intelligible to the people who have to live it. Process redesign, the deliberate restructuring of the workflow to capture the Capacity Flip, creates the operational substrate for the third order. Encoded judgment, the architectural layer, is where sovereignty is built.

Invert the order and each intervention misfires. Invest in tools before redesigning the work and the tools optimise the wrong output. Redesign the work before the people understand what they are replacing and the redesign happens around a process no one is equipped to govern. Encode judgment into infrastructure the institution does not control and the sovereignty never materialises. The Three-Pillar Model is not a menu.

Three Diagnostic Questions

Three questions, one per order, let any institution locate itself on this path:

1. Can professionals critically evaluate and improve AI-generated output, or do they accept it uncritically? (1st order · Education)
2. Have workflows been redesigned around human-AI complementarity, or is AI bolted onto existing processes? (2nd order · Process Redesign)
3. Is proprietary knowledge encoded in sovereign infrastructure the institution controls, or does it live inside generic vendor solutions? (3rd order · Encoded Judgment)

Each order, tackled alone, fails in a predictable way.

What This Means

The productivity dashboard is real. It is also insufficient. The institutions that will matter in five years are not the ones with the highest Copilot adoption curves today. They are the ones that have already absorbed what the Capacity Flip requires of their roles, and that have already begun the architectural work of encoding what they know into infrastructure they control.

Architectural parity is the condition. Encoded judgment is the response. Sovereignty is the outcome. Anything short of that is first-order optimisation in strategic clothing, which is the Copilot Fallacy in three words.

The mistake I see most often is the belief that the first order is where transformation is happening, because the first order is where it is visible. The metric that is easiest to produce is almost never the metric that matters. Second- and third-order effects do not announce themselves on a quarterly basis. They show up, slowly, in which institutions are still relevant in the next decade and which ones are not.