D2 3D Swim Tracking

Topic

3D swimming tracking is a technique that reconstructs the three-dimensional trajectories of individual fish—or groups of schooling fish—from image sequences captured simultaneously by multiple synchronized cameras, each viewing the fish from a different perspective. The physical principle underlying this reconstruction is stereoscopic triangulation: the 3D position of a point on the fish's body is determined by intersecting the optical rays that define the projection of that point onto at least two calibrated cameras. This process utilizes the geometry relating the images from both cameras—known as epipolar geometry—along with the projection matrices obtained for each camera during a prior calibration phase using a target with known point positions.

When 3D swimming tracking is applied to a school of fish rather than a single individual, the reconstruction process also faces an assignment problem: for each pair of consecutive frames, the technique must match each detection at the current time step to the corresponding fish's identity from the previous step. This assignment is resolved by minimizing a global cost function that combines positional distance with consistency regarding each fish's velocity. To achieve an optimal assignment—even in the presence of detection noise—a Kalman filter is employed; it predicts each fish's future position based on its estimated velocity and subsequently corrects that prediction using the new detection, assigning greater or lesser weight to either the prediction or the measurement depending on which is more reliable at that moment. Once reconstructed, these three-dimensional trajectories allow for the direct calculation of kinematic swimming parameters for each fish—such as speed, acceleration, turning angle, and the radius of the curve traced. In the case of a school, they also enable the calculation of the group's overall kinematic parameters: the degree of schooling alignment, the distribution of relative speeds between neighbors, and the functions describing the correlation between a fish's movement and that of nearby fish. By combining these 3D trajectories with accelerometer measurements, it is also possible to estimate the fish's sinuosity and cost of transport directly in its natural environment, eliminating the need for the controlled conditions of a laboratory flume.

The primary advantage of 3D trajectory reconstruction is that it reveals the discrepancy between the fish's net path—the straight line connecting its starting and ending points—and the actual path it traveled; this difference goes undetected in 2D tracking, which ignores changes in depth. This distinction is particularly significant for species that undergo substantial depth changes while swimming; in such cases, 3D swimming tracking—described earlier as a technique for reconstructing three-dimensional trajectories—becomes essential to avoid underestimating the extent and nature of the fish's actual movement.

ID:2486