Practice

Automation is a change of task, not a removal of one

Automating a step does not leave a person with one fewer thing to do. It leaves them with a different thing to do, usually monitoring, which is the task people perform worst.

Bainbridge set out the ironies in 1983: automation takes the tasks a designer knew how to specify and leaves the operator those that resisted specification, which are the harder ones; it then asks that operator to monitor, which people do poorly; and it removes the routine practice by which the skills needed for intervention were maintained. Skill decay after a year of nonuse has since been measured at d = −1.4. The residual task is not the old task with a step removed.

A feature that performs a step automatically is described, accurately enough, as saving the person that step. What the description omits is the step that replaces it. Where the system acts, somebody must establish that it acted correctly, decide when it has not, and be capable of doing the thing themselves when it fails. These are three tasks, and the literature on them is forty years old and largely unread outside its own field.

Bainbridge's paper in Automatica remains the clearest statement because it is structured as a set of consequences rather than a warning. The automated system handles the situations its designer anticipated; the situations left to the person are the ones the designer could not specify, which is to say the unusual, the ambiguous and the compound. The person is then placed in a monitoring role, and vigilance research had already established that sustained monitoring for rare events is among the activities human attention supports least well. Meanwhile the routine instances that would have maintained the person's competence are the ones the automation has taken, so the skill required at the moment of intervention is decaying continuously between interventions.

The decay is quantified. Arthur, Bennett, Stanush and McNelly's 1998 meta-analysis in Human Performance drew 189 data points from 53 articles on trained skills subsequently unused, and found an effect size for decay of d = −0.01 immediately after training, rising to d = −1.4 after more than 365 days of nonuse. Cognitive tasks decayed faster than physical ones. The person asked to take over from a well-functioning automation is, by construction, somebody who has not performed the task recently.

Where the task went

Parasuraman, Sheridan and Wickens proposed in 2000 a framework that makes the relocation visible. Automation can be applied at four stages of human information processing — acquiring information, analysing it, deciding, and implementing the action — and at varying levels within each stage. A feature described as "automating the task" has in practice automated one or two stages and left the others. A system that gathers and summarises but does not decide has moved the person's work from acquisition to evaluation. A system that decides and acts has moved it to supervision.

The framework's value is that it names what was left. Evaluation and supervision are not residual; they are demanding tasks with their own failure modes, and they arrive without the practice that the automated stages used to supply.

Parasuraman and Riley's 1997 review in Human Factors classified what then goes wrong. Misuse is overreliance on automation that is performing within its design envelope but outside the conditions the person assumes. Disuse is rejection of automation whose false alarms have exceeded tolerance. Abuse is automating a function because it can be automated, without analysis of the consequences for the person left in the loop. The third is the most common in product work and the least remarked upon, because it does not look like a decision at the time it is taken.

The current instance

Features that generate a draft, classify an item, or propose a value have the structure Bainbridge described, with one aggravating property: the output is fluent. A proposed value that is wrong does not announce itself by looking wrong, which means the evaluation task the person has been handed is harder than the task that was taken away. Composing a paragraph requires knowing what it should say; checking a paragraph that reads well requires knowing what it should say and resisting the plausibility of the text already on the screen.

Where the automation is right most of the time, the evaluation task also loses its own reinforcement. A check that finds nothing on ninety-nine occasions is a check performed with progressively less attention on the hundredth, which is the vigilance finding restated in a product setting.

What the analysis consists of

Allocation of function is the method, and it predates computing: for each function in the task, a deliberate decision about whether it is performed by the person, by the system, or by both, recorded with the reason. What makes it a method rather than an intention is that it asks the question the other way around as well. For every function given to the system, what is now required of the person, what does that require them to know, and how do they come to know it if the system is doing the work.

The answers constrain design. Where the person must detect an incorrect output, the system has to expose what the output was derived from, because detection without access to the inputs is not possible. Where the person must take over, the handover has to begin before the failure is total, since a person brought in at the point of breakdown has neither context nor warm skill.

The two paths

The practical change is that the automated path and the manual path are designed as one system rather than as a feature and a fallback. The derivation of an automatic result is visible beside the result, so that checking is a comparison rather than an act of faith. Confidence is expressed as a property of the particular output rather than of the feature, so that the cases warranting attention are distinguishable from the cases that do not. And the manual route stays present and practised rather than being archived behind a setting, because a route nobody has taken for a year is a route that exists in the interface and not in the person.

Sources

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. doi:10.1016/0005-1098(83)90046-8 - Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics — Part A, 30(3), 286–297. Source of the four-stage framework. - Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. Source of the misuse/disuse/abuse classification. - Arthur, W., Bennett, W., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative review and analysis. Human Performance, 11(1), 57–101. doi:10.1207/s15327043hup1101_3. Effect sizes and study counts as reported. - Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32–64.

All sources are paraphrased; nothing above is quoted. The application to generative features is the author's extension of Bainbridge's argument, not a finding reported in any cited source. The vigilance claim is general to that literature and is stated without a figure for that reason.

← All writing