Practice

Migration toward the boundary is a normal process, not a deviation

Workarounds are not indiscipline. Rasmussen described the pressures that move any working system toward the edge of what it can tolerate, and they operate whether or not anyone intends them to.

Rasmussen's 1997 model describes an operating point pushed by two gradients: management pressure toward efficiency, and individual pressure toward least effort. Both move the system toward the boundary of acceptable performance, and neither is a failure of anybody's intentions. Behaviour a product team records as misuse is the visible part of that migration, and it is the best available evidence about where the boundary now sits.

Every deployed product accumulates practices its designers did not specify. Data is maintained in a spreadsheet beside the system because the system's fields do not fit the case. A credential is shared because the permission model has no role matching the job somebody actually does. A record is created with placeholder values because the form demands fields that are not yet known and the work cannot wait. A step is skipped because it has never once produced a different outcome.

The comfortable reading is that these are violations: individually chosen, correctable by instruction, and attributable to the people performing them. Rasmussen's 1997 paper in Safety Science set out why that reading survives so poorly. He described a system operating within a space bounded on several sides — by economic failure, by unacceptable workload, by functional acceptability — and subject to systematic pressures. Management presses toward efficiency, which moves the operating point away from the economic boundary and toward the others. Individuals press toward least effort, which moves it further. Neither pressure is pathological; both are the normal conduct of a working organisation, and together they produce a drift toward the boundary of acceptable performance that nobody authored.

The consequence Rasmussen drew is the one that matters here. Accidents do not generally arise from a single decision to behave unsafely. They arise when a system that has migrated steadily toward its boundary encounters a variation it would once have absorbed. The migration was the condition; the variation was only the occasion.

The same shape in product work

Snook's 2000 study of the shootdown of two American helicopters over Iraq described a related mechanism he called practical drift: locally sensible adaptations, each rational within the unit that made it, accumulating into a global state no one had designed and no one could see. The units had not disobeyed. They had adapted to the conditions they actually faced, and the adaptations were incompatible with each other in a way that became apparent only on the day it mattered.

Product systems drift the same way, with the same local rationality. The spreadsheet exists because it is faster than the system for the case at hand. The shared credential exists because the alternative is a request that takes four days. Each adaptation is defensible in isolation, and the aggregate is a working practice the product does not model, cannot see, and does not protect.

Cook and Rasmussen's 2005 account in Quality and Safety in Health Care added the property that makes this hard to detect. A system under sustained efficiency pressure operates with progressively less margin, and the loss of margin is invisible while nothing goes wrong. Performance looks unchanged until the point where it does not, because margin is what gets consumed first and margin is not measured.

What the behaviour is evidence of

Treating a workaround as misconduct discards information. The adaptation is a measurement of the distance between the task as specified and the task as performed, taken by the person best placed to take it and at their own expense. It identifies where the design has failed to accommodate a real case, how frequently that case arises, and what the people doing the work consider the cost of compliance to be.

Reason's distinction between active failures and latent conditions locates the design responsibility. The latent conditions are built in when the system is designed and lie dormant until circumstances combine with them. A permission model that cannot express a role the organisation actually has is a latent condition. It does not cause anything on the day it ships. It causes the shared credential eighteen months later, and the sharing is an active failure only in the sense that it is the part that happened last.

The observable margin

The change is to treat adaptation as a signal the product is obliged to receive. Where a workaround is known, the design question is which constraint produced it and whether that constraint was load-bearing. Many are not: a mandatory field that is routinely filled with placeholder text was never collecting data, and making it optional with an explicit unknown state recovers both the record and the truth about it.

Where the constraint is load-bearing, the adaptation still establishes that the compliant path costs more than people will pay, and the remedy is to lower that cost rather than to raise enforcement. A permission request that resolves in minutes rather than days removes the reason for the shared credential more reliably than a policy forbidding sharing.

The structural version of this is to design the margin to be observable. A system that records how often a step is bypassed, how often a record is created incomplete, and how often an action is reversed is measuring its own distance from the boundary while that distance still exists. Without that instrumentation, the first report that the margin has gone is the incident that consumed it.

Sources

Rasmussen, J. (1997). Risk management in a dynamic society: A modelling problem. Safety Science, 27(2–3), 183–213. doi:10.1016/S0925-7535(97)00052-0. Source of the operating-point model and the two gradients. - Cook, R., & Rasmussen, J. (2005). "Going solid": A model of system dynamics and consequences for patient safety. Quality and Safety in Health Care, 14(2), 130–134. doi:10.1136/qshc.2003.009530. Source of the account of margin consumed under efficiency pressure. - Snook, S. A. (2000). Friendly Fire: The Accidental Shootdown of U.S. Black Hawks over Northern Iraq. Princeton University Press. Source of practical drift. - Reason, J. (1990). Human Error. Cambridge University Press. Source of the active failure / latent condition distinction. - Reason, J. (1997). Managing the Risks of Organizational Accidents. Ashgate.

All sources are paraphrased; nothing above is quoted. The product examples — the parallel spreadsheet, the shared credential, the placeholder record — are the author's illustrations of the cited mechanisms and are not drawn from the sources.

← All writing