E42 Multisensory fusion

Topic

Multisensory fusion combines information from different channels—such as vision, proprioception, and the vestibular system—to produce an estimate of the state of the world that is more accurate than one obtained from any single channel alone. The Bayesian framework establishes that optimal combination involves weighting each signal according to its uncertainty: the lower the variance (and thus the higher the precision), the greater the weight assigned in the final estimate.

Combining multiple independent sensors progressively improves precision. For identical sensors, the gain in precision increases with the square root of the number of sensors. When two sensors with the same individual variance are combined, the resulting estimate has half the variance. In general, increasing the number of independent sources reduces the uncertainty of the combined estimate. This principle also applies when the modalities have different levels of precision: the most reliable signal receives the most weight, yet the less precise signal can still contribute useful information. According to the text, the human nervous system approximates this optimal behavior in position-estimation tasks that combine vision and tactile input from the hand.

However, before fusing two signals, it is necessary to determine whether they stem from the same cause. Two sensory stimuli may represent different aspects of a single object or event, or they may originate from independent sources. A causal Bayesian model estimates the probability of a common cause and adjusts the degree of integration accordingly. When this probability is high, the signals are strongly fused; when it is low, they are processed more independently.

This principle explains the ventriloquism illusion. When a visual signal and an auditory signal appear sufficiently close together, the system may attribute them to a common cause and shift the perceived location of the sound toward the visual source. If the spatial separation becomes too great, the probability of a common cause decreases, and integration begins to fade. Consequently, the greater the disparity between the two signals, the weaker their tendency to fuse. The text links this mechanism to other Bayesian inference processes regarding the body, such as the rubber hand illusion.

The McGurk effect is another example of multisensory integration. Hearing a specific speech sound while observing a visual articulation corresponding to a different one can give rise to the perception of a third phoneme. According to the described model, the resulting percept corresponds to the one offering the greatest joint likelihood when auditory and visual information are considered simultaneously. Neither modality alone fully explains the result; rather, it emerges from their integration based on the assumption that both signals originate from the same speech source.

The same principle of uncertainty-based weighting appears in the Kalman filter. Its gain determines the extent to which the organism should rely on the prediction provided by its internal model versus a new sensory measurement. If the newly received measurement is highly precise, the estimate shifts primarily toward it. Conversely, if the internal prediction has less uncertainty than the observation, the prediction is given greater weight. Weighting thus continuously adapts to the relative reliability of both sources.

This mechanism also constitutes a fundamental principle of inertial navigation systems, where accumulated internal estimates must be periodically corrected via external observations. According to the text, the cerebellum implements a comparable adaptive weighting process within sensorimotor coordination, complementing mechanisms analogous to the Kalman filter for state estimation and the LQR controller for action control.

Overall, multisensory integration does not simply consist of summing signals. First, it must be determined whether they correspond to a common cause, and then each must be weighted according to its precision. Causal Bayesian inference determines when integration is appropriate, while principles such as Kalman gain establish the degree of reliance on each source and on internal predictions. The nervous system thus obtains more precise and robust estimates, continuously adapting integration based on the uncertainty and coherence of the available information.

ID:2398