A Preference Is Not a Value

A person clicks the short video instead of the long essay. A recommender records the click. A manager accepts the suggested schedule. A customer keeps the default. Each event is evidence of a choice. None is a complete statement of what the person values.

The distinction sounds philosophical until a system begins optimizing. Then it becomes operational. If a platform treats every pause as desire, every purchase as endorsement and every repeated habit as identity, it can turn a narrow observation into a durable model of the person. The model then changes what the person sees, making the next observed preference partly a response to the system's previous guess.

A preference is an observation under conditions

Suppose someone chooses A over B. The result can support a modest claim: in that moment, with those options, costs, descriptions, defaults, time limits and social pressures, A won. Change the conditions and the result may change. Add a third option, remove a fee, delay the decision, explain a hidden consequence or let the person confer with someone else, and the ordering can reverse.

Economics often uses revealed preference to infer an ordering from behavior. That can be useful for prediction. It does not mean the action reveals a timeless inner essence. The inference depends on what was available, what the person knew and which constraints were binding. A commuter may choose the fastest route while valuing clean air. A parent may buy the convenient meal while valuing nutrition. A worker may accept a meeting time while valuing uninterrupted concentration. Constraint is not hypocrisy.

Sourced fact: NIST's AI Risk Management Framework treats context of use, affected individuals and human judgment as necessary parts of risk management. Inference: behavior captured without those conditions is weak evidence for a broad claim about a person's enduring values.

Values are not merely stronger preferences

A preference can be immediate, local and easily traded. A value can organize choices across time, impose duties toward other people and remain important even when acting on it is inconvenient. The categories overlap, but they are not interchangeable.

People also hold plural values that conflict. Privacy can compete with convenience. Loyalty can compete with candor. Rest can compete with achievement. A single action settles one encounter between those commitments; it does not prove the loser was never valued. Treating the winning action as the whole person erases the conflict that made the decision meaningful.

This is why a profile built from behavioral data should be described as a model of observed patterns, not a moral biography. It can estimate what someone may click next. It cannot, from the click alone, determine what that person believes should matter.

Interfaces manufacture part of the evidence

Choices do not arrive in neutral containers. Position, color, timing, defaults, interruption and the cost of declining all change behavior. The essay The Default Is a Decision Someone Else Made examines one powerful case: the preselected option. A default can reduce friction for a beneficial action, exploit inattention or do both for different users.

Recommendation systems add another layer. They decide which alternatives become visible before measuring which visible alternative wins. A song cannot be preferred in the logged data if it was never offered. A news source can disappear from a person's apparent interests because ranking made it costly to reach. The measurement instrument edits the field it measures.

Stable conclusion: optimization based on behavioral signals is partly self-referential. It should be evaluated not only by short-term prediction accuracy but also by the opportunity structure it creates.

Prediction can become preference production

A system predicts that a user likes sensational material, shows more of it and receives more sensational clicks. The new data appears to confirm the original prediction. Yet exposure was not independent. The system helped produce the behavior it later treats as validation.

This does not require manipulation in the dramatic sense. Small selection effects accumulate. A feed that repeatedly chooses the easiest-to-measure response can train both the model and the user toward that response. The problem in The Scoreboards We Mistake for Life applies here: once a measure controls distribution, participants adapt to the measure.

The proper question is not whether the preference is real. The click happened. The question is how much of the pattern belongs to prior disposition, current context, interface design, learned habit and the narrowing effect of earlier recommendations. Usually the log cannot separate them by itself.

Prediction is not permission

Even an accurate prediction does not grant unlimited authority to act on it. Knowing that someone is likely to accept a default is not permission to make cancellation obscure. Inferring that a person may be distressed is not permission to target vulnerability. Predicting a child's attention is not equivalent to the child's informed consent.

The OECD AI Principles call for human rights, democratic values, transparency, robustness and accountability across the AI lifecycle. Those are governance commitments, not properties that emerge from higher click-through rates. A system can predict well and still use the prediction badly.

Separate three questions:

1. Evidence: What behavior was observed, under which conditions?
2. Inference: What limited prediction does the evidence support, for how long?
3. Authority: What may the system or organization legitimately do with that prediction?

Collapsing the third question into the first is how descriptive data quietly becomes a license.

A humane model must leave room for revision

People change, experiment and contradict themselves. That is not noise to be eliminated. Revision is part of agency. A profile that continually resurrects an old pattern can make change expensive: the recovering gambler sees betting promotions, the person leaving a political identity sees only material that reinforces it, or the student who once avoided mathematics is never offered a demanding course.

Useful systems need forgetting, reset, correction and exploration. A person should be able to inspect important inferences, remove stale ones, choose a neutral starting point and receive options outside the learned pattern. The Personal Algorithm Audit provides a user-side protocol; organizations need equivalent controls in product design and governance.

Use the preference-to-value test

Before translating behavior into a claim about a person, run seven checks:

1. Option set: What alternatives were genuinely available and visible?
2. Constraint: Which price, time, access, disability or social pressures shaped the choice?
3. Knowledge: What material facts could the person reasonably know?
4. Interface: Which defaults, rankings or interruptions altered the decision?
5. Horizon: Does this signal predict only the next action, or something more durable?
6. Conflict: Which other stated commitments may have lost in this instance?
7. Revision: How can the person correct, reset or outgrow the inference?

If those questions cannot be answered, lower the claim. Say “recently clicked,” not “cares about.” Say “predicted to choose,” not “is the kind of person who.” Precision here is not politeness. It is epistemic hygiene.

Design for declared aims and observable behavior

Stated goals also have limits. People can misremember, perform an identity or want incompatible outcomes. The answer is not to crown statements and ignore behavior. It is to hold multiple forms of evidence without pretending they are interchangeable.

For consequential systems, ask people about the outcome they are trying to achieve, observe whether the system helps, and allow them to change the aim. Measure long-horizon welfare and guardrails alongside engagement. Preserve alternatives that do not maximize the immediate signal. Document whose objective function is operating: the user's, the institution's or the advertiser's.

This approach accepts that a person is not a clean utility function waiting to be extracted. A model can still be useful. It becomes more useful when its claims remain narrower than the human being.

The bottom line

A preference is real evidence, but it is evidence of a choice in a situation. A value is a wider commitment interpreted across situations, conflicts and time. Systems fail when they mistake the first for a complete account of the second—and become dangerous when they treat prediction as permission.

Record behavior precisely. Interpret it modestly. Give people a way to revise the model.

Research and framework notes

• NIST, AI Risk Management Framework, including context-of-use and human-judgment guidance.
• NIST AI Resource Center, AI risks and trustworthiness characteristics.
• OECD.AI, OECD AI Principles.
• For adjacent analysis, read Algorithmic Reality and Friction Is a Feature.