This presentation requires a current web browser.

Adapt or Drift: Resilience Engineering When AI Moves the Operating Point

Andrew Hatch

The Blocker Lemming represents deliberate safety intervention: pause or redirect work to preserve margin.

Session: Adapt or Drift: Resilience Engineering When AI Moves the Operating Point.

We have more capability than ever before

But less certainty we’re in control

Decades of research on sociotechnical systems and complexity still explain these risks.

Will our ability to notice, understand and correct keep pace?

AI is changing what we can do and how fast we can do it. Rasmussen’s model still helps us understand the pressures. The question is whether our ability to notice, understand and correct can keep pace.

AI can radically change how work gets done while still fitting familiar patterns of adaptation under pressure. But it can also influence what people accept as evidence and how intent becomes action. That changes the mechanisms we use to understand and control the system.

The framework can remain useful while our assumptions need rechecking. Decades of research give us a lens. They don’t guarantee that familiar review processes or feedback loops will remain adequate.

Rasmussen supplies the boundary and control framework. Applying it to AI’s tempo, verification demand and feedback delay is this talk’s synthesis. Evaluate whether people, tools and workflow retain the capacity to intervene and recover. AI can also improve assurance.

Sources:

WHOAMI

Australia
United States
Cisco ThousandEyes
  • Tester
  • Lead Tester
  • Business Analyst
  • Developer
  • Lead Developer
  • Technical Architect
  • Software Engineering
  • SharePoint Developer
  • Solution Architect / Solutions Architect
  • .NET Technical Architect
  • Integration Consultant
  • Integration Architect
  • SharePoint 2007 Consultant
  • BizTalk Server Consultant
  • .NET Architecture Consultant
  • TFS Configuration Management Lead
  • ERP Integration Team Lead
  • Microsoft Office Solution Consultant
  • Configuration Management Lead
  • Solution Architect – Information Management and Portal
  • Solution Architect – .NET Web Application
  • DevOps Engineer
  • Data Migration
  • Web Development
  • DevOps Architect
  • .NET Developer
  • Senior IT Consultant
  • Head of Platform Engineering

SWE/SRE Engineering Manager 15 years.

AWS
Data centre
LinkedIn
Medium Bluesky

@hatchman76

Introduce your background:

  • Australia and the United States.
  • Cisco ThousandEyes. SWE/SRE Engineering Manager, 15 years.
  • AWS and data centres.
  • LinkedIn.
  • Medium and Bluesky: @hatchman76.

Takeaways

Models: not always accurate, but they are useful....
⚠️ This model is a system-wide lens focused on risk, built from localized analysis, actions, and perspectives.
🌱 It is a framework for understanding the conditions that lead to adversity and how to stay resilient and adaptive to change.
👩🏿👨🏼🧑🏽🧕🏻 Because we all work and live within complex socio-technical systems.
🔥 And any attempts to change a complex system, either in whole, or in part, can lead to unintended consequences.

Some recurring characters

AI, LLM, and agent symbols equals

Anything AI!
LLM, Agent etc...

A complex system sign equals

any complex distributed
software system.

Introduce the recurring images as visual shorthand the audience will see throughout the talk and should keep in mind. The AI image includes LLMs, agents and related tools. The complex-system image represents a complex distributed software system, of which those tools are a part.

Socio Technical Systems

A socio-technical system cannot be understood by examining either people or technology in isolation.

The visual contrasts studying people and technology separately with seeing people as part of the system. Its connections emphasise the interactions between them.

This names the system the talk will examine and prepares Rasmussen’s control hierarchy.

People are part of control

People and organisations are adaptive components and controllers within the socio-technical system, not an external human input. Locally sensible adaptations can preserve control, or interact to create conditions for failure. Rasmussen connects effective control with usable objectives, feedback, competence, coordination and commitment.

Source: Rasmussen (1997), Figures 1 and 3, control, adaptation and information.

Resilience Engineering

Build and sustain adaptive capacity in socio-technical systems, both technical and non-technical, ensuring they can absorb and recover safely from incidents, anomalies and changing pressures.

Build the ability to adapt before it is needed

The opening definition is this talk’s synthesis, not a quotation. Resilience Engineering is the study and deliberate support of the capacities that help socio-technical systems continue to perform as conditions change. Sustaining that ability means maintaining expertise, coordination, time, resources and authority to act, and renewing them as the system evolves. Adaptive capacity is not simply spare infrastructure capacity.

Incidents and anomalies make adaptation visible, but it also occurs in everyday work. Keeping functioning can involve changing plans, priorities or ways of working. This does not promise immunity from disruption or unlimited capacity to absorb any event. Prepare for situations that exceed what was anticipated, and learn how people already prevent difficulties from becoming worse.

John Allspaw’s March 6, 2019 presentation abstract also places investment in adaptive capacity at the centre of the field. It remains a background source for the definition: Amplifying Sources of Resilience.

Surprise and human adaptation

David D. Woods: “Resilience concerns the capabilities a system needs to respond to inevitable surprises.” Resilience is a Verb (2018), p. 1.

Richard I. Cook: “Human practitioners are the adaptable element of complex systems.” How Complex Systems Fail, point 12, p. 3 (1998, revised April 21, 2000).

Portrait of Jens Rasmussen

Jens Rasmussen
Human Factors and Safety Science

Rasmussen's original dynamic safety model

Risk Management in a Dynamic Society (1997)

Radiological safety Maritime safety Healthcare safety

Socio-Technical Systems

- Hierarchical

- Multi-disciplinary

Systems Thinking

Introduce Jens Rasmussen and the 1997 model. The next slide maps the wider research landscape around Figure 3, and the slide after that locates people and technology within Rasmussen’s social control hierarchy before the boundary model is translated into plain language.

  • The original model comes from Risk Management in a Dynamic Society (1997).
  • Examples of safety-critical domains include radiological safety, maritime safety and healthcare.
  • Socio-technical systems are hierarchical and multidisciplinary.
  • Systems thinking connects these levels and disciplines.

Paper: Rasmussen, Risk Management in a Dynamic Society (1997).

The research behind the model

The 31-page paper connects decades of studies across engineering and the human sciences.

Page 201 of Rasmussen's paper, showing four research traditions converging on models of behaviour-shaping work features and criteria
Figure 8: research traditions converge

Risk analysis Different hazards require different strategies

Defence in depth Redundant layers can erode without visible failure

Human performance under pressure Studies move from error rates to work in context

Feedback control Controllers need current state, authority, and competence

Naturalistic decisions Local adaptations interact across the system

This is a synthesis paper rather than a single empirical validation of the model.

  • Rasmussen distinguishes empirical, evolutionary, and analytical risk strategies.
  • Defence in depth matters, but redundant protections can degrade quietly because a local violation may have no immediate visible effect.
  • The paper reviews human reliability, operator modelling, field evidence, and organisational research. Its direction of travel is away from isolated error counts and toward behaviour in a real work context shaped by workload, time, cost, and competing goals.
  • Risk management becomes a feedback-control problem spanning the whole socio-technical hierarchy.
  • Decisions are embedded in continuous work. Locally sensible adaptations can interact across controllers and create system-level risk.

The next slide makes the paper’s system-wide control hierarchy concrete, and the following slide simplifies the geometry without discarding these mechanisms.

Source: Rasmussen, Risk Management in a Dynamic Society (1997), pp. 183–213.

  • Figure 8 and research convergence: p. 201.
  • Risk strategies: pp. 197–200.
  • Defence in depth: pp. 189–190, 198–199, 207–209.
  • Human and organisational adaptation under pressure: pp. 186–190, 193–196, 206–209.
  • Feedback control: pp. 183–186, 192–197.
  • Naturalistic decision making: pp. 186–190, 200–206.

The social control hierarchy surrounds the people and technology doing the work. Research disciplines and environmental stressors cross its levels.

Control is distributed rather than owned by any single actor, and information must travel upward while decisions and constraints travel downward. Each layer adapts on a different timescale.

Goals, evidence and authority

The control structure is the distributed network of decision makers, goals, constraints, information, competence and authority around the work. Observations and outcomes inform decisions, and decisions and constraints change the work. Feedback must reach actors with the resources and authority to respond.

When AI enters these channels, re-examine problem framing, attention, verification, skill retention under pressure, communicated intent and feedback fidelity across the hierarchy. Task-level studies do not establish the accumulated effects on a whole socio-technical system.

Source: Rasmussen (1997), Figures 1 and 3, control, adaptation and information.

AI, LLM, and agent symbols in a colored ball
...and how AI changes the conditions to adapt.

The hierarchy on the previous slide surrounds the people and interactions in this complex system. People and technology operate within economic, workload and performance constraints. The dotted error margin marks a buffer inside the performance boundary.

The following slide explains the boundaries and the margin in detail.

Adaptation with LLM tools

Tools change the conditions in which people work, and repeated adaptations change workflows, expectations and later decisions. Rasmussen remains a framework for examining these effects. Measures of human performance, learning, coordination and control need revalidation in LLM-mediated work. This does not make earlier human-factors research obsolete.

Source: Rasmussen (1997), Figures 1 and 3, control, adaptation and information.

Boundary to economic failure

Boundary to unacceptable workload

Boundary of functionally acceptable performance

A dotted line inside the actual boundary Error Margin
Operating Point Red circle marking the operating point
Playing Surface
Cost efficiency Least effort

Plain-language script

“The economic boundary marks where we can no longer pay the bills. The workload boundary marks where people cannot keep going. The performance boundary marks where the system can no longer do its job acceptably or safely. We cannot see that line exactly, so the dotted line gives us a margin for error.

This dot is not a person. It is the whole system — people, software, and rules — working right now. The dim dot marks the starting reference position, not a second current operating state. The shaded area is the playing surface, with the error margin shown in orange.

Economic and workload limits can change as financial and human capacity change. Inward shifts reduce the playing surface; outward shifts expand it.

Pressure for efficiency and less effort can draw normal adaptations, and the whole system's operating state, toward the performance boundary.

The charts stand for pressure to reduce cost, which a downturn can intensify. The hands stand for ways people reduce effort under workload; the Blocker Lemmings show safety action near the error margin: pause or redirect work to preserve margin. Rasmussen describes actors adapting within the work-space, encountering an effort gradient while management supplies a cost gradient. The charts and hands are scattered through that space to illustrate local adaptations, not measured locations of individual decisions in Figure 3.

The economic-side arrow shows the gradient toward cost efficiency; the workload-side arrow shows the gradient toward least effort. Both point toward the acceptable-performance side. Figure 3 and its adjacent discussion support these directions of migration. The arrow endpoints are illustrative rather than measured positions or fixed targets.

Everyone makes sensible choices. The business asks for cheaper and faster. People make work easier. Together, those pressures can move the dot toward the performance edge.

There is no magic safe spot in the middle. The lines do not push the dot back. Any line can move as demand, staffing, technology, or AI changes.

That is Rasmussen's warning: failure does not require one stupid decision. Normal adaptations can slowly move the system toward an edge nobody can see exactly.”

Core point: one current operating state moves within economic, workload, and acceptable-performance constraints; pressures move it, while sensing and control preserve margin and recoverability.

Sources

  • https://doi.org/10.1016/S0925-7535(97)00052-0
  • https://backend.orbit.dtu.dk/ws/files/158016663/SAFESCI.PDF (Figure 3 and its discussion of effort and cost gradients)
  • https://srecon24.hatchman76.com/#/the-model
  • https://qconsf.com/

Diagram Legend

Economic boundaryWhere money runs out

Workload boundaryWhere people cannot keep up

Performance boundaryWhere the system stops working acceptably

Error marginBuffer before the performance limit

Operating pointWhere the whole system is now

Playing surfaceRoom within the boundaries

Local Actions and External pressures Change the operating point or its constraints.

Recap the symbols used in the Fish diagram. The operating point represents the current state of the whole socio-technical system, not a person. The dotted line marks a perceived limit inside the actual performance boundary; the gap between them is the error margin. Boundaries constrain the playing surface, but they do not push the point in any direction. The combined Blocker Lemming, hand and chart are this talk's shorthand for actions and pressures: safety interventions, adaptations to reduce effort, and pressure toward cost efficiency. Rasmussen's Figure 3 describes local adaptive variations, effort and cost gradients, and a safety countergradient. Figure 1 shows technological, market and political stressors acting across the socio-technical hierarchy. Local actions and wider pressures interact; an external condition is not necessarily a deliberate action. In this adaptation, changes can shift the operating point or reshape the constraints around it. The Blocker Lemming means pause or redirect work to preserve margin. It represents an active intervention by people with evidence, authority and capacity; the boundary itself does not stop the point. Actions may arise within a team's immediate work or elsewhere in the control hierarchy. Management and regulators remain part of the wider socio-technical system; environmental stressors act across it. Source: Rasmussen, Risk Management in a Dynamic Society (1997), Figures 1 and 3 and the discussion of migration toward the boundary and control of system performance. https://backend.orbit.dtu.dk/ws/portalfiles/portal/158016663/SAFESCI.PDF

Economic Boundary

External conditions change what the business can sustain.

More room to operate

  • Stronger demand and better pricing power
  • Fresh funding or cheaper financing
  • Lower energy, supplier or infrastructure costs

Less room to operate

  • Market slowdown, churn or price competition
  • Costlier credit or funding that dries up
  • Higher supplier costs or currency shocks

Greater financial capacity expands room to operate, tighter budgets increase pressure to reduce costs.

These are common external conditions, not measurements of a universal boundary. Holding the work and commitments broadly constant, stronger sustainable revenue, accessible capital or a lower sustainable cost base can expand financial viability. Losing them can contract it. Favorable demand also brings work: growth does not automatically increase every margin. An internal cost target or budget cut is a management decision; it is not itself the actual economic-failure constraint. Watch actual cash, obligations and value.

Activity, capacity and control

More consumption or commitments can move the operating state toward economic failure when costs outpace value and supporting financial capacity. Sustainable savings can expand viable operating space; reduced revenue, funding or runway can contract it. Forecasts provide feedback, and budgets or quotas are management targets and decision thresholds. Revised targets do not by themselves change the actual viability constraint. Rasmussen draws the perceived boundary and error margin only on the performance side.

📉

Economic Boundary Actions

Signals become decisions that push toward cost efficiency.

Leading signal / metricPossible management action
Forecast cash burn increases
Reduce scope, defer investment,
reprioritize commitments
Cost per verified outcome increases
Renegotiate suppliers, rightsize resources,
remove low-value work
Idle capacity or unused licences
Consolidate platforms, retire unused capacity
Projected margin decreases
Freeze hiring, reduce staffing or maintenance

Cutting waste can reduce costs, but cutting expertise or maintenance can leave less room to operate safely.

A forecast is a signal and a budget is a target. An actor with authority decides what to do. These are possible organizational responses, not recommendations to reduce staffing or resilience. Unit economics should include human verification and recovery. Right-sizing and retiring unused services can preserve performance; cutting necessary expertise, maintenance or redundancy while retaining commitments can consume performance margin. Validate the effect of savings across the whole workflow. Existing incidents or realized losses are lagging outcomes; forecast cash and projected margins are earlier financial signals.

Account for transferred work

Chapter 5 §5.2.4 (printed p. 81; supplied PDF p. 89) asks who receives a saving, who absorbs the work, and when it becomes visible. Track authoring savings beside reviewer, maintainer and on-call costs. Section 5.5.2 (printed p. 85; PDF p. 93) extends evaluation to future adaptability, coupling, congestion and conflict. A local saving is not sufficient evidence of a net workflow saving.

Financial feedback must reach people with authority, competence and resources to change commitments. They must communicate effects on review demand, maintenance and recovery across levels. Savings preserve other margins when low-value work is removed. If output stays fixed while expertise or review are cut, the resulting adaptations can transfer pressure into workload and acceptable performance.

  • Rasmussen (1997), Figure 3; printed manuscript pp. 9–12, 15–18 and 27 (not journal pagination)
  • Rayo, Morey, Grøtan and Woods (2026), AI Gold Rush 2 Versus AI Realism, Chapter 5

Workload Boundary

The sustainable capacity to do and understand the work can change.

More room to operate

  • More trained people and relevant expertise
  • Better tools and reliable toil reduction
  • Protected focus, recovery time and knowledge sharing

Less room to operate

  • Attrition, layoffs or unavailable specialists
  • Fatigue, burnout and constant interruptions
  • Fragile tooling and lost operational knowledge

Less capacity moves the boundary inward, more work can push the operating point toward it.

Holding the kind and quality of required work broadly constant, usable expert capacity can change the sustainable workload limit. Headcount is an imperfect proxy: new hires need onboarding, relevant expertise, tools and coordination. Greater raw demand is not itself a capacity expansion; it can instead move the operating point toward the workload constraint. Automation can remove toil but adds maintenance and exception work. Burnout and lost knowledge can reduce future effective capacity. These examples are mechanisms for discussion, not a numerical mapping from staff count to diagram geometry.

Focused verification capacity

Effective capacity means relevant expertise, focused time and enough context for this work, not merely assigned headcount. A reviewer can be allocated to many tasks yet lack usable time to assess this change. Protect contextual knowledge and recovery; changing demand or removing expert capacity can also change the verification effort required per item.

  • Rasmussen (1997), Figures 1 and 3; control, adaptation and information
🫷

Workload Boundary Actions

People adapt toward least effort when demand strains capacity.

Early signal / metricPossible adaptation
Queue age and work in progress
Automate repeatable toil, improve tooling,
limit incoming work
Interruptions and handoffs
Focus time
Batch tasks, standardize paths,
simplify tools and coordination
Overtime and fatigue
Recovery time
Reduce scope, protect rest, share on-call load
or rush and bypass checks
Review findings and rework
Improve tests and shared understanding
or accept output without examining it

Reducing toil can preserve control, but skipping checks saves effort now and creates work later.

The hand denotes the gradient toward least effort, not a moral judgment. Better technology and automation can remove repeatable toil and improve assurance. They require maintenance, competence and exception handling. Under pressure, equally understandable adaptations can thin verification or normalize rushed handoffs. Queue age, interruptions and recovery time can warn of emerging capacity problems. Review findings/rework are current quality feedback and possible warnings about future workload, not universally leading indicators. Burnout is a condition to investigate with people, not a diagnosis from a single metric. These actions do not guarantee a particular trajectory; watch their effect on verification and recovery.

Match admission to assurance

Count admitted work, focused verification effort per change and effective capacity. The review-load ratio in TALK.md is this talk’s illustrative capacity heuristic, not a measure established by Chapter 5. Demand rising relative to capacity can increase backlog and delay. Defects additionally depend on adaptations such as reduced review or testing; queue growth alone does not demonstrate a defect increase.

Change scope, timing or admitted work, and accept latency when necessary. Preserve source evidence and give reviewers time and practical authority to reject, pause or reverse action. Chapter 5 §5.1 (printed pp. 78–80; PDF pp. 86–88) illustrates responsibility without resources through a scenario, not a measured universal outcome. AI can strengthen tests, diagnosis and verification where tools actually help.

  • Rasmussen (1997), Figures 1 and 3; control, adaptation and information
  • Rayo, Morey, Grøtan and Woods (2026), AI Gold Rush 2 Versus AI Realism, Chapter 5

Boundary of
Acceptable Performance

Crossing the functional limit means unacceptable outcomes.

Failure in the service

  • Outages and cascading system failures
  • Lost, corrupted or exposed customer data
  • Recovery that cannot restore service in time

Consequences

  • Customer operations disrupted, safety put at risk
  • Trust and reputation damaged
  • Contractual losses or regulatory consequences

Can people, tools and workflows anticipate and adapt to failure, ensuring system capacity remains resilient to performance degradation and recovers?

The acceptable-performance boundary is defined by the system's required functions and tolerated harms. Not every crossing is an immediate catastrophe: an unacceptable outcome can be local or escalate through dependencies. Availability, data integrity, confidentiality and safe recovery are software examples. Reputation, commercial losses and regulatory consequences can follow that failure; they are not separate Rasmussen boundaries. Formal SLOs and compliance checks are useful proxies, not an exact picture of the true functional limit. Crossing the dotted perceived boundary enters the protected error margin; it does not prove a failure has already occurred. The model does not provide a measurable distance to catastrophe.

Joint performance over the lifecycle

Evaluate people, tools and workflow together across good, mediocre and poor AI output. A model benchmark alone cannot establish operational joint performance. Chapter 5 §5.5.3 (printed pp. 85–86; PDF pp. 93–94) calls for evaluation situated in the actual work and repeated through the lifecycle.

Compare verified outcomes, revision and checking effort with a relevant baseline. Include a plausible consequential mistake in a controlled exercise: can people access the evidence, detect it and correct it before enactment? Measure outcome quality, verification effort, consequential-error detection, response time and successful recovery. Re-evaluate when models, workflow, staffing or dependencies materially change. These are proposed engineering measures rather than a universal score; operational signals remain imperfect proxies for the true functional limit.

Morey, Rayo and Woods (2025) studied predictive clinical AI in simulated patient cases with 450 nursing students and 12 licensed nurses. Recommendation correctness improved or degraded joint performance. Added explanations did not significantly change those effects in this experiment. It motivates the evaluation design; it is not an LLM software-risk estimate or evidence that explanations never help.

  • Rayo, Morey, Grøtan and Woods (2026), AI Gold Rush 2 Versus AI Realism, Chapter 5
  • Morey, Rayo and Woods (2025), npj Digital Medicine, Results and Discussion

The Error Margin

Notice, act and recover before the functional limit is crossed.

Signal / concernProtective actions
Blind spots or stale telemetry
Improve monitoring, check user journeys, connect alerts to an owner who can act
Saturation or shrinking headroom
Improve software efficiency, load-test, add scalable capacity and tested redundancy
Changes exceed verification capacity
Limit admission, use canaries and rollback, retain independent review and access controls
Slow or uncertain recovery
Exercise failover and restore, update runbooks, practice diagnosis and share expertise

Sustain adaptive capacity by turning signals into effective action

The dotted line represents a fallible perceived limit. Crossing it consumes the deliberately protected error margin; failure is not inevitable at that point. An alert does not push the operating point back. It needs timely interpretation, authority to slow or change work and people with tools to execute a response. Capacity, efficiency and redundancy may improve feasible performance, but need realistic load, failure and recovery testing: correlated dependencies can defeat nominal redundancy. Canaries, least privilege, independent review, rollback and restore exercises preserve bounded excursions and recovery. An error budget is a service-management mechanism, not a geometric measurement of Rasmussen's error margin.

Notice, intervene and recover

Deliberate interventions pause or redirect work to preserve margin. Someone needs the evidence, authority and capacity to act.

Notice requires usable signals and timely interpretation. Intervention requires evidence, authority, tools and remaining response capacity. Recovery requires contextual knowledge and fallbacks that people can use under pressure. A written rollback procedure does not establish that capability. Awareness and the margin itself exert no automatic restoring force; the objective is sufficient margin, bounded excursions and recoverability, not a preferred geometric center.

Exercise stale observations, scarce time, excess review demand and unavailable supporting services outside production. Assess whether the team can pause, reduce scope and recover. Chapter 5 §§5.3.2–5.3.5 (printed pp. 82–84; PDF pp. 90–92) discusses difficulties recognizing plausible errors; §5.1 illustrates responsibility without resources and decay of alternatives; §§5.4–5.5 (printed pp. 84–86; PDF pp. 92–94) support challenging exercises and continuing assurance.

The AWS 19–20 October 2025 disruption report describes conventional automation and dependencies, with no reported generative AI involvement. Delayed DNS-plan application interacted with changed state and cleanup; subsequent EC2 recovery entered congestive collapse. AWS announced recovery testing and controls on incoming work. The report does not establish that all announced work was complete. Use it as a control-and-recovery mechanism example, not an AI-caused incident or a general verdict on automation.

  • Rasmussen (1997), Figures 1 and 3; control, adaptation and information
  • Rayo, Morey, Grøtan and Woods (2026), AI Gold Rush 2 Versus AI Realism, Chapter 5
  • AWS, 19–20 October 2025 disruption: DynamoDB, EC2 and corrective actions
Red circle representing the whole system’s current operating state
The coloured AI ball rolls in from the left and covers the operating point

What happens as AI infiltrates every part of the system?

The operating point represents the whole socio-technical system's current state. The AI artwork is shorthand for AI-mediated work at that same state, not a second operating point, a person or a fourth constraint. The reference ball marks the starting state, not an ideal or inherently safe location.

Ask what changes in cost, effort, evidence and recovery when AI enters the work.

AI in the feedback and action channels

AI is not automatically a new controller. It influences channels used by existing controllers. On the feedback side it can gather and summarize observations, find contradictions and support testing; fluent summaries can also conceal uncertainty. On the action side it can present options, draft decisions and translate intent into output; plausible detail can exceed the communicator’s intended criteria.

AI output changes the operating state when people or automation admit and enact the resulting decisions, not merely when output is generated. The risk is conditional: AI can strengthen testing, diagnosis, challenge and verification. Risk rises when generation and action become easier faster than assurance and correction.

Rasmussen (1997) describes behaviour and work activity migrating toward the boundary. Cook and Rasmussen (2005) explicitly use operating-point terminology. The ball remains shorthand for one current whole-system state.

  • Rasmussen (1997), Figures 1 and 3; control, adaptation and information
  • Cook and Rasmussen (2005), Going solid

Boundary to economic failure

Boundary to unacceptable workload

Boundary of functionally acceptable performance

And work function and outputs rely heavily on non-determinism?

AI ball representing the current operating point

The same constraints still apply

Economic failure, unacceptable workload and functionally unacceptable performance remain the three constraints. The error margin is the protected performance-side buffer, not an additional constraint. AI does not create a fourth boundary.

The moving ball represents one current whole-system state. The reference ball marks the starting position, not a second current state or an ideal geometric centre.

The trajectory illustrates how adaptation under pressure can move the operating state toward the performance side. It is not a measured path. Organisational choices and usable feedback shape the actual trajectory, and no geometric centre guarantees safety.

Ask how non-deterministic outputs change what people can understand, verify and rely on. Examine the consequences for economics, workload, expertise and the capacity to intervene and recover.

📉

AI has to be paid for

Cheaper generation can still mean higher total cost.

The bill grows

  • 47% of surveyed tech leaders reported AI spending over plan. ETR, 2026
  • Uber’s annual AI budget was exhausted by April. CTO interview, 2026
  • More seats, agent loops and adoption add demand.

Money moves

  • Cut infrastructure refresh to fund AI overruns: 49% of firms reallocating budgets. ETR, 2026
  • Bring in outside capital: Meta’s $27bn Hyperion venture with Blue Owl. 2025
  • Redirect investment toward AI products with stronger expected returns.

Savings here can add work elsewhere. Who covers it and at what cost?

ETR surveyed 1,636 technology leaders, June 2–July 5, 2026. Its 47% is reported spending above plan, not a prevalence estimate of annual budgets exhausted early. Among respondents reallocating to cover overruns, 61% cut contractor/consultant spending, 49% legacy infrastructure modernization or hardware refresh, 36% non-AI licenses and 28% internal headcount budgets. The subgroup denominator matters; a smaller headcount budget does not itself prove layoffs. The slide's 49% applies only to organizations reallocating budgets to cover AI overruns, not all surveyed organizations. The Uber case is an executive's reported account rather than audited finances. Meta's October 2025 Hyperion announcement describes a joint venture with Blue Owl funds to finance roughly $27 billion of data-center development. This brings in outside capital rather than cutting another internal budget, with future lease and guarantee obligations. The amount is project development cost, not Meta's annual AI bill. The portfolio-return example is an illustrative management choice. These cases distinguish reduced infrastructure investment, external financing and product reprioritization. Cost control can preserve other margins by dropping low-value work. It transfers pressure when required output stays constant but relevant expertise, maintenance or assurance disappears. More AI expenditure moves the operating state; budgets remain targets, not an extra dotted economic boundary.

Whole-workflow economics

Include model use, platform commitments, integration, tools and data, evaluation, human verification, rework and recovery in outcome cost. Longer contexts, retries, agent loops and additional workflows can consume unit-cost savings. This is conditional, not a prediction that cheaper AI must increase spending. Evaluate work transferred or deferred to reviewers, maintainers and responders, along with future coupling, congestion and recovery costs.

Preserve capacity by reducing low-value demand, bounding loops, changing scope and routing or caching where verified quality permits. Organizations may also redirect portfolios toward products with stronger expected returns, as the visible example suggests. Keep verification and recovery funded. AI can improve tests and diagnosis as well as generation. If required output stays constant while relevant expertise or review is removed, workload can rise and performance margin can shrink.

Rasmussen supplies the adaptation/control framework; Chapter 5 supplies a contemporary resilience synthesis. The software economics and response examples are this talk’s application, not AI findings from the 1997 paper. Economic Chapter 5 references: §5.2.4, printed p. 81/PDF p. 89; §5.5.2, printed p. 85/PDF p. 93.

  • Rasmussen (1997), Figure 3; printed manuscript pp. 9–12, 15–18 and 27 (not journal pagination)
  • Rayo, Morey, Grøtan and Woods (2026), AI Gold Rush 2 Versus AI Realism, Chapter 5

Tokenomics Evolution

StageControl intent
Maxxing Treat token consumption as a score to maximise.
Metering Count token use to track limits and allowances.
Optimization Route users’ requests to cheaper models to reduce costs.

How does commercial AI evolve from here, and what are the risks?

Token consumption, dependence and platform power

Tokenmaxxing turns LLM-agent consumption into a public scoreboard. The tokenmaxxing.sh leaderboard ranks token counts and estimated API-equivalent spending. Those estimates are not reconciled bills, and token consumption does not measure useful outcomes, productivity or quality. These are language-model usage tokens, not cryptocurrency. The site was checked on October 27, 2026.

Doctorow’s enshittification model describes two-sided platforms. They first offer users substantial value, then shift value toward business customers such as advertisers and sellers, then squeeze both groups to retain more surplus for the platform. Business customers are distinct from the platform owner. This is a conditional analogy about AI dependence and pricing power, not a claim about every provider or measured model quality.

Cursor’s August 12, 2025 announcement describes a change in metering. Individual Auto usage had been unlimited since June. The announcement said it would count against included monthly usage at token rates from each user’s next renewal after September 15, 2025. Some annual renewals could fall in 2026. This dates the announcement and planned transition, not independently confirmed rollout for every subscriber. This does not establish deliberate lock-in, reduced model quality or rent extraction.

Optimization describes a business routing users’ requests to cheaper models to reduce costs. Microsoft’s model-router documentation describes a Cost mode that favours smaller, cheaper models, with possible quality trade-offs on complex prompts. Routing requires evaluation against the actual workload. Cheaper routing alone does not demonstrate deliberate degradation or Doctorow’s full platform cycle.

The parallel is a risk to examine: when workflows depend on a provider, that provider controls the meter and can change the terms. Token leaderboards can reward consumption while the organization needs useful, verified outcomes. These documented examples do not establish that every AI provider has completed Doctorow’s cycle. Ask about switching costs, portability, alternative suppliers and outcome cost before increased use becomes a commitment that is difficult to unwind.

  • Cory Doctorow, Enshittification (MCD, 2025), publisher page and excerpt: https://us.macmillan.com/books/9780374619329/enshittification/
  • Doctorow, original platform lifecycle essay (January 21, 2023): https://pluralistic.net/2023/01/21/potemkin-ai/#hey-guys
  • Tokenmaxxing public leaderboard: https://tokenmaxxing.sh/
  • Michael Truell, Cursor pricing announcement (August 12, 2025): https://cursor.com/blog/aug-2025-pricing
  • Microsoft Learn, How model router works (checked October 27, 2026): https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router-how-it-works

Model Enshittification

Serve Users

Make it useful and easy to adopt.

Favor Business

Put paying business customers first.

Squeeze for Rent

Charge more. Give both sides less.

AI prices have fallen for the same capability. But!!

Stanford AI Index, 2025

Lower costs don’t guarantee lower prices.

Cloud-market example: CMA, 2025

Cory Doctorow, Enshittification (2025)

Doctorow describes a two-sided platform moving value from users to business customers, then taking more from both groups. Business customers mean advertisers, sellers or other participants, distinct from the platform owner. The headings are a paraphrase. Applying this pattern to AI is a risk to examine, not a prediction that every provider follows it.

Lower costs do not determine who keeps the savings

Stanford’s AI Index article reports that the price of querying models at GPT-3.5-level MMLU performance fell from $20 per million tokens in November 2022 to $0.07 in October 2024. Its 2025 report also describes hardware costs per unit of performance declining by 30% annually. These are historical trends, not guarantees about future token prices or total workflow costs.

The cloud analogy concerns competition and switching. For its UK and EEA market definition, the CMA’s July 2025 cloud decision found concentrated supply, sustained AWS and Microsoft returns above their cost of capital, and barriers to switching. Its April 2024 foundation-model review identified risks from control over critical inputs and routes to customers. These findings support questioning how fully efficiency gains reach customers, rather than asserting an inevitable AI oligopoly.

The October 2026 CMA update records improvements, including removal of transfer fees for switching providers, while identifying remaining barriers. Cloud prices have risen for some services and fallen for others. The precedent is imperfect competition, not a claim that cloud prices never fall.

DeepSeek still supplies MIT-licensed model weights and charges for its hosted API. Open weights do not mean free compute. The evidence does not establish that incumbents will push DeepSeek out. The question is whether competing models remain practically accessible and switching remains viable as the market evolves.

🫷

AI and least effort

AI can make “generate, accept, act” the path of least effort.

Proxy the judgment

Pass off plausible AI reasoning as your own judgment.
Quick answers can hide a lack of understanding.

Meat-proxy emoji: a person represented by a steak passing messages between two AI bots

Multiply the streams

Agents generate more proposals, and humans inherit
review, integration and maintenance.

A bucket overflowing with AI slop

Squeeze recovery

Extra checking and context switching can
raise workload and burnout risk.

🤯
Workslop is polished-looking output that leaves the recipient to reconstruct missing thinking. The BetterUp/Stanford survey of 1,150 full-time US desk workers found 40% reporting receipt in the prior month, and roughly two hours on average to resolve an incident. These are self-reports, not an objective audit of every AI-generated artifact. The chain is this talk's software-workflow illustration: a person may use an agent to proxy an argument, answer or decision without forming their own understanding. The recipient must check the assumptions and reconcile conflicts. AI can also reduce this effort when used to build comprehension and assurance. Berkeley researchers observed eight months at one 200-person US technology company and conducted more than 40 interviews. Broader task scope, parallel streams and work extending into breaks/evenings intensified work. This motivates burnout risk from reduced recovery, not a universal causal finding. Expertise, familiarity, staffing and the design of handoffs affect who carries the added load; these sources do not prove that lower-skill or less-parallel users invariably suffer more burnout. Completed backlog counts should be read alongside verification time, rework, interruptions, recovery and ability to explain changes. The hand depicts least-effort adaptation. The meat-proxy emoji illustrates an engineer coping with increased workload by relaying plausible AI reasoning, arguments or judgments as their own, without understanding the fundamentals. This can create the appearance of expertise while bypassing the work that develops and exercises it. It can also leave the recipient to verify reasoning the sender cannot explain. Keep a person accountable for understanding, explaining and checking the work they pass onward.

Evidence, capacity and understanding

Least effort describes normal adaptation under competing goals and limited attention, not a moral claim about laziness. Plausible ≠ measured. Fluent ≠ intended. Fast generation does not establish fast verification, integration or maintenance. Conflicting AI proposals still require people to reconstruct context and understand fundamentals. The meat-proxy scenario is this talk’s illustration, not a claim that all AI use reduces expertise. Chapter 5 §§5.3.1–5.3.4 (printed pp. 82–83; PDF pp. 90–91) discusses filtering, revision and plausible errors that need expertise to recognize and correct.

Current LLM evidence shows changes in task pace and cognitive effort that differ by task and experience. Lee et al. (CHI 2025) surveyed 319 knowledge workers: critical-thinking effort shifted toward verification, response integration and task stewardship, while higher confidence in AI was associated with less scrutiny. This is self-reported association, not causal proof. Dell’Acqua et al. randomized 758 consultants: work was faster and higher quality within the model’s capability frontier, but correctness was lower outside it. These are task-specific results, not a universal effect or a whole-system risk estimate. Dillon et al. and Brynjolfsson, Li and Raymond provide additional heterogeneous workplace evidence.

Preserve focused attention, contextual knowledge and usable authority to change commitments. Change scope, timing or admitted demand when checking cannot keep pace. AI can improve assurance capacity where joint evidence supports its testing and diagnosis. Demand relative to capacity can increase backlog and delay; defects additionally depend on adaptations that shed assurance. Do not treat generated backlog clearance as verified completion.

  • Rayo, Morey, Grøtan and Woods (2026), AI Gold Rush 2 Versus AI Realism, Chapter 5
  • Lee et al. (CHI 2025), The Impact of Generative AI on Critical Thinking
  • Dell’Acqua et al., Navigating the Jagged Technological Frontier
  • Dillon et al. (2025), Shifting Work Patterns with Generative AI
  • Brynjolfsson, Li and Raymond (2025), Generative AI at Work

The erosion of expertise

Accelerated Expertise: Training for High Proficiency in a Complex World, by Robert R. Hoffman and colleagues
Hoffman et al. (2013).
Accelerated Expertise.
Psychology Press.

“However, all of our lessons learned in cognitive systems engineering and complexity theory imply that more automation may result in simplifications, but they will hide deeper complexity, and at those moments where resilience and high-performance are needed, there will have to be greater operator expertise.”

The Reverse Centaur’s Guide to Life After AI, by Cory Doctorow

... or worse

Cory Doctorow (2026), The Reverse Centaur’s Guide to Life After AI.

Accelerated Expertise: Training for High Proficiency in a Complex World, by Robert R. Hoffman, Paul Ward, Paul J. Feltovich, Lia DiBello, Stephen M. Fiore and Dee H. Andrews.

Citation year and publisher follow the publisher's DOI record: 2013, Psychology Press, DOI 10.4324/9780203797327. The print edition carries a 2014 copyright year.

The supplied quotation is attributed to Accelerated Expertise. Use it to discuss how automation can simplify the surface of work while obscuring complexity that demands greater expertise when resilience and high performance are needed. This is a discussion of the risk to expertise, not a measured claim that all AI use erodes it.

Doctorow’s reverse-centaur distinction asks who controls the work: whether a person chooses how a machine assists them or is required to serve the machine’s pace and demands.

Cory Doctorow, The Reverse Centaur’s Guide to Life After AI (MCD, 2026): https://us.macmillan.com/books/9780374621568/thereversecentaursguidetolifeafterai/

Book-cover image supplied by Macmillan: https://mpd-biblio-covers.imgix.net/9780374621568.jpg

The people, shrugging, flame and “This is Fine” are a metaphor for uncertain understanding and unaddressed trouble in a complex system. The system remains complex even when the people responsible for it can no longer explain what is happening or how to intervene.

This is an illustration of a risk to expertise and control, not evidence that every use of AI causes expertise loss or failure.

Artwork credit: This is Fine by KC Green, animated by Philcrifo. Original comic: https://gunshowcomic.com/648. Animation credit: https://imgur.com/gallery/i-animated-this-is-fine-comic-tjxrc.

where we think we are

Toad screaming in panic
Your SREs!

The buckets represent accumulating verification and repair work. The Lemming represents the expertise and response capacity needed to protect performance. Cheap generation can move checking, maintenance and recovery work elsewhere, consuming attention and capacity even as visible output increases.

The dotted line warns about the remaining buffer. Crossing it begins consuming protected error margin, while the solid line marks the actual functional limit. A warning helps only if people have practical understanding, time, authority and means to reduce demand, challenge output, stop further action or recover.

The faded reference marks where we think we are, while the moving point illustrates how actual conditions can drift. This is an illustrative possible journey, not a measured trajectory or claim that all AI output is workslop. Organisational choices and usable feedback shape whether people can detect and respond to a convincing mistake in their actual work.

GIF source: Toad Screaming, shared by BrendaMcCoy on Tenor.

Skills that protect margin

Expertise turns warnings into protective action.

Rasmussen directs attention to how locally reasonable decisions interact across the whole system. AI can change cost, effort and the pace of proposed action. Expertise helps people see those interactions, question plausible explanations and recognise when a plan no longer fits. Protecting margin requires useful feedback and the capacity to act on it. Crossing the dotted line begins consuming the protected buffer before the actual functional boundary. It does not establish that harm has already occurred.

Sources: Rasmussen (1997), Cook (2000), Hoffman et al. (2013), Woods (2018).

Learning Opportunities, by Dr. Cat Hicks, offers optional 10–15 minute exercises using prediction, explanation and reflection during AI-assisted coding. Use it to keep active learning in engineering work. GitHub repository.

Tool vs Operator

Test people, tools and workflow together.

AI can assist diagnosis, testing and explanation, and can also be authorised to act. It can change existing feedback and action channels. The distinction changes what must be bounded and observed. People need useful evidence, practical understanding and enforceable authority to interrupt or recover. Evaluate the joint workflow with relevant baselines and both helpful and misleading outputs. The clinical joint-performance research motivates this question but does not estimate software or LLM risk. Whether an application helps depends on its use and evaluation, not the label “tool” or “operator”.

Sources: Rasmussen (1997), Morey et al. (2025), Rayo et al. (2026), Chapter 5.

We have to adapt

  • Economic: Count generation, verification, operation and switching costs.

  • Workload: Match AI output to the team’s ability to understand and check it.

  • Performance: Test the work, including plausible AI mistakes and recovery.

  • Error margin: Give people time, expertise and authority to turn signals into action.

Homer Simpson looking from side to side

Local savings can create pressure elsewhere.

The same three constraints remain: economic viability, acceptable workload and acceptable performance. AI can save effort in one place while moving verification, maintenance or recovery work elsewhere. Count the whole workflow and protect the people who interpret warning signals and own corrective action. Error margin is a protected performance-side buffer, not an SLO error budget or a fourth constraint. These proposed responses need local evidence and adjustment as conditions change.

Sources: Rasmussen (1997), Rayo et al. (2026), CMA (2025).

Homer side-eye GIF reused from the SREcon26 presentation: https://srecon26.hatchman76.com/images/homer-simpson-side-eye.gif

Humans are adaptable

Expertise, coordination and judgment help us respond when plans no longer fit.

Use AI to strengthen our capacity to understand, act and recover.

Adaptive capacity is developed before surprise and renewed through everyday work. Expertise, shared understanding and coordination help people change plans when familiar responses are no longer enough. They also need time, resources and authority to act. AI can contribute to that capacity when its use supports understanding and effective action. The aim is sustained joint performance and recoverability. This is the talk's application of resilience engineering, not a guarantee that humans or tools can absorb unlimited pressure.

Sources: Cook (2000), Allspaw (2019), Woods (2018), Hoffman et al. (2013).

References

Model & operations

AI research

Work & budgets

Thank you!
https://qcon26.hatchman76.com