Practice

A confirmation prompt tests attention, not intention

Twenty-five million warnings were measured in the field. The best-performing ones stopped most people; the worst were bypassed seven times in ten. What none of them could do was tell a correct decision from a confident mistake.

Akhawe and Felt instrumented Chrome and Firefox and recorded 25,405,944 warning impressions. Clickthrough on malware and phishing warnings ran between nine and twenty-three per cent; on Firefox's SSL warning, thirty-three per cent; on Chrome's, 70.2 per cent. The spread shows that warning design changes behaviour substantially. It also marks the ceiling: every one of those warnings was asking a question only an inattentive person can fail.

A confirmation prompt interrupts an action and asks whether the person meant it. The design intent is to catch the case where they did not. What the prompt can actually detect is narrower than that, and the difference decides whether it works.

Two failures lead to an unwanted action. In the first, the person intended something else and the action departed from the intention — a misdirected click, a wrong row, a slip in Reason's terminology, where the plan was sound and the execution was not. Against this failure a prompt performs respectably: the interruption breaks the automatic sequence, the action is named back, and a person who reads the name recognises the mismatch. In the second, the person intended exactly this action, on a belief about the system or the data that happens to be wrong. Reason calls the result a mistake, and the check a prompt imposes has already been performed internally, to the person's satisfaction, before the prompt appeared. Premature closure describes the mechanism: the search for alternatives has terminated, and the prompt invites a review the person has no reason to conduct.

The prompt therefore samples attention. It asks whether the person is present enough to notice a discrepancy, and returns a useful answer where a discrepancy exists to be noticed. Where the discrepancy is between the world and the person's model of it, the prompt has no access to either.

What the field data shows

Akhawe and Felt's 2013 study for the USENIX Security Symposium used in-browser telemetry from Chrome and Firefox to count warning impressions and the proportion bypassed, across 25,405,944 impressions in May and June of that year. Malware and phishing warnings were clicked through between nine and twenty-three per cent of the time. Firefox's SSL warning was bypassed in 33.0 per cent of cases. Chrome's SSL warning was bypassed in 70.2 per cent.

The result is usually cited for the high figure and the conclusion drawn that warnings do not work. The authors drew close to the opposite conclusion, and it is the more useful one: the variation across warnings is large, which establishes that design has substantial purchase on the outcome. A warning that stops nine people in ten is not the same artefact as one that stops three.

The variation also indicates what the design is working on. Firefox required three clicks to bypass and permitted stored exceptions; Chrome required one. Clickthrough tracked the effort and the frequency of exposure rather than anything about the correctness of the underlying decision. Almuhimedi and colleagues at Google found in 2014 that clickthrough on Chrome's malware warning varied with the person's browsing history, which is a statement about exposure, not about judgement.

Both findings point the same way. The dependent variable these designs move is how much attention the interruption commands. Where attention is commanded and the action was a slip, the slip is caught. Where attention is commanded and the action was a mistake, the person attends, confirms, and proceeds.

Habituation

Exposure degrades the mechanism in a direction that cannot be designed around within the prompt. A warning displayed on every occurrence of a condition, most of which are benign, trains the response that clears it. The response becomes part of the action sequence rather than an interruption to it, which is the point at which the prompt has stopped sampling attention and started measuring nothing at all.

This gives the deployment of prompts a property that distinguishes it from most interface decisions: each additional prompt reduces the effectiveness of the others. A product with one confirmation at its single irreversible step has an instrument. A product with eleven has a ritual.

The controls that act on intention

Checks that survive a confident mistake share a structure, in that they require information the mistaken belief does not supply. Independent re-entry is the clearest case. Asking a person to type the name of the item being deleted, rather than to confirm that deletion is intended, requires them to look at what is actually selected. The belief that the right row is selected does not generate the characters; the row does. The check has moved from attention to evidence.

Delay with reversal is the second. Where the action executes but remains undoable for a period, the person who was confidently wrong receives what a prompt could never give them: the consequence, visible, while the correction is still cheap. The error surfaces through its effect rather than through anticipation of it.

Separation is the third and the least used. Where the destructive action is not adjacent to the routine one, is not reachable by the same gesture, and is not the default position of a menu, the slip stops occurring, and the prompt that guarded against it becomes unnecessary.

Placement

The question for any existing prompt is which failure it was placed there to catch. Where the answer is a slip, the prompt is doing work that separation would do permanently and without spending attention. Where the answer is a mistake, the prompt was never the right instrument, and independent re-entry or a reversal window is.

The practical effect in a product is a smaller number of interruptions and a different kind: confirmation disappears from the routine path, the irreversible step requires something that only the correct state can supply, and the actions that can be undone are undoable from the place where their consequences appear.

Sources

Akhawe, D., & Felt, A. P. (2013). Alice in Warningland: A large-scale field study of browser security warning effectiveness. Proceedings of the 22nd USENIX Security Symposium, 257–272. Impression count, clickthrough ranges and the 33.0 and 70.2 per cent SSL figures are as reported there, as is the observation about click counts and stored exceptions. - Almuhimedi, H., Felt, A. P., Reeder, R. W., & Consolvo, S. (2014). Your reputation precedes you: History, reputation, and the Chrome malware warning. Proceedings of the Tenth Symposium on Usable Privacy and Security (SOUPS). - Reason, J. (1990). Human Error. Cambridge University Press. Source of the slip/mistake distinction. - Reason, J. (2000). Human error: Models and management. BMJ, 320(7237), 768–770. doi:10.1136/bmj.320.7237.768 - ISO 9241-110:2020, Ergonomics of human-system interaction — Part 110: Interaction principles, 5.6.2 (use error avoidance) and 5.6.4 (use error recovery).

All sources are paraphrased; nothing above is quoted. The account of habituation reflects the general warning-science literature rather than a specific quantitative result, and is stated without a figure for that reason. The characterisation of prompt counts in deployed products is the author's observation.

← All writing