E45 Decision making under uncertainty

Topic

Decision-making under uncertainty involves selecting an action when the outcomes cannot be fully predicted. Expected utility theory provides the normative framework for this process: the optimal action is the one that maximizes the sum of the utilities of the various possible outcomes, weighted by their respective probabilities. Real organisms, however, systematically deviate from this theoretical optimum, and these deviations offer insight into the internal mechanisms governing their decisions.

The drift-diffusion model provides a mechanistic description of how a decision is reached and elaborates on the principle of evidence accumulation. Sensory information accumulates progressively at a drift rate determined by the signal-to-noise ratio until a specific decision threshold is reached. This threshold establishes a trade-off between speed and accuracy: the higher the threshold, the more evidence must be accumulated, resulting in a decision that is slower but also more accurate. According to the text, neurons in the parietal lobe and a specialized visual area exhibit increasing activity consistent with this accumulation process.

Hyperbolic temporal discounting describes how the value assigned to a reward diminishes as the reward is delayed further into the future. Unlike exponential discounting, it gives rise to time-inconsistent preferences. Two future rewards may initially hold a specific preference ranking, but this ranking can reverse as time passes and one of the rewards becomes more immediate. This behavior is observed in humans as well as in rats, pigeons, and non-human primates. Thus, the temporal proximity of a reward can significantly alter its relative attractiveness compared to a more distant reward.

The reward prediction error signal constitutes a central mechanism of temporal reinforcement learning. Midbrain dopaminergic neurons respond to the difference between the expected reward and the reward actually received. An unexpected reward elicits a positive response, whereas activity decreases when an expected reward fails to materialize; ...and when the result matches the prediction exactly, there is virtually no error signal. The greater the surprise caused by the reward relative to prior expectations, the stronger this signal becomes. In this way, the organism can progressively update its predictions and modify future decisions based on experience.

Attitude toward risk introduces another deviation from the simple normative model. Because the marginal utility of a reward diminishes as its magnitude increases, an organism may prefer a certain reward over an uncertain alternative with the exact same expected value. This risk aversion implies that an additional gain yields progressively less subjective utility, giving certainty a value of its own.

However, this attitude shifts when the decision is framed in terms of losses. According to the prospect theory described in the text, an organism tends to be risk-averse when facing potential gains but may become risk-seeking when attempting to avoid a loss. The valuation of gains and losses is therefore asymmetric: losing a resource can have different subjective consequences than obtaining an equivalent gain. The text also suggests a possible evolutionary value for this loss aversion in environments where resources fluctuate and retaining what is available may be particularly important.

Taken together, these mechanisms demonstrate that decision-making under uncertainty involves far more than simply comparing probabilities. Expected utility provides the normative framework for choosing between uncertain outcomes; the drift-diffusion model explains how evidence accumulates until a decision is reached; hyperbolic discounting incorporates the temporal distance of rewards; and prediction error enables learning from the discrepancy between expectations and actual events.

The final decision thus emerges from the interplay of probability, utility, available evidence, time, learning, and risk. Organisms may systematically deviate from the normative optimum through preferences for immediate rewards, risk aversion regarding gains, or risk-seeking behavior regarding losses. These deviations are not merely random errors, but regular patterns that reveal how the organism adapts its decisions to uncertainty, its prior experience, and the possible consequences of each action.

ID:2377