Practice

Automation is a change of task, not a removal of one

Automating a step does not leave a person with one fewer thing to do. It leaves them with a different thing to do, usually monitoring, which is the task people perform worst.

Bainbridge set out the ironies in 1983: automation takes the tasks a designer knew how to specify and leaves the operator those that resisted specification, which are the harder ones; it then asks that operator to monitor, which people do poorly; and it removes the routine practice by which the skills needed for intervention were maintained. Skill decay after a year of nonuse has since been measured at d = −1.4. The residual task is not the old task with a step removed.

A feature that performs a step automatically is described, accurately enough, as saving the person that step. What the description omits is the step replacing it. Where the system acts, somebody must establish that it acted correctly, decide when it has not, and be capable of doing the thing themselves when it fails.

Bainbridge's paper remains the clearest statement because it is structured as consequences rather than warning. The automated system handles what its designer anticipated; what is left to the person is what the designer could not specify, which is to say the unusual, the ambiguous and the compound. The person is then placed in a monitoring role, and vigilance research had already established that sustained monitoring for rare events is among the activities human attention supports least well. Meanwhile the routine instances that would have maintained competence are precisely the ones the automation has taken. Arthur and colleagues put a number on the consequence: decay rising to d = −1.4 after more than a year of nonuse, faster for cognitive tasks than physical ones. The person asked to take over from a well-functioning automation is, by construction, somebody who has not done the task recently.

Parasuraman, Sheridan and Wickens make the relocation visible. Automation can be applied at four stages — acquiring information, analysing it, deciding, implementing — and at varying levels within each. A feature described as automating the task has usually automated one or two stages and left the others. A system that gathers and summarises but does not decide has moved the person's work from acquisition to evaluation; one that decides and acts has moved it to supervision. Neither is residual. Both are demanding tasks with their own failure modes, arriving without the practice the automated stages used to supply.

Features that generate a draft, classify an item or propose a value have this structure with one aggravating property: the output is fluent. A proposed value that is wrong does not announce itself by looking wrong, so the evaluation task handed over is harder than the task taken away. Composing a paragraph requires knowing what it should say; checking a paragraph that reads well requires knowing what it should say and resisting the plausibility of the text already on the screen. Where the automation is right most of the time, the check loses its own reinforcement too — a check that finds nothing on ninety-nine occasions is performed with less attention on the hundredth.

Allocation of function is the method, and it asks the question both ways: for every function given to the system, what is now required of the person, what must they know, and how do they come to know it if the system is doing the work. The answers constrain the design. Detection of a wrong output is impossible without access to what it was derived from. Handover has to begin before failure is total, since somebody brought in at breakdown has neither context nor warm skill.

Sources

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.

Parasuraman, R., Sheridan, T. B., and Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics — Part A, 30(3), 286–297. Source of the four-stage framework.

Arthur, W., Bennett, W., Stanush, P. L., and McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative review and analysis. Human Performance, 11(1), 57–101. Effect sizes as reported.

Parasuraman, R., and Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.

All sources are paraphrased; nothing above is quoted. The application to generative features extends Bainbridge's argument rather than reporting a finding from any cited source. The vigilance claim is general to that literature and is stated without a figure for that reason.

← All writing