Revisiting Multi-View Stereo: A Sequence-to-Sequence Formulation

Abstract

Computing accurate geometry from multi-view images is a fundamental problem in computer vision. Recent feed-forward (FF) models jointly estimate 3D geometry and camera parameters, but they typically suffer from geometry distortion caused by reconstruction ambiguity, even when ground-truth camera parameters are supplied. In this paper, we study the multi-view stereo (MVS) problem with known camera parameters and propose a novel approach that bridges conventional MVS and FF methods. Rather than casting MVS as a sequence-to-one mapping that predicts depth only for a single reference view, we reformulate it as a sequence-to-sequence task, akin to FF models, that jointly predicts geometry for all input views. We introduce a global transformer-based architecture with two components that explicitly exploit camera-induced priors: ray-map embeddings that inject camera parameters into image patch tokens, making the transformer camera-aware, and a unified global cost volume that replaces conventional per-view cost volumes to jointly capture 3D structure across all views. Extensive experiments on multiple public benchmarks show our approach achieves state-of-the-art performance, surpassing both MVS and FF reconstruction baselines.

Video

Sequence-to-Sequence Formulation

Sequence-to-one MVS, sequence-to-sequence MVS and feed-forward reconstruction.

Conventional MVS methods follow a sequence-to-one formulation: a view-specific cost volume is built for a single reference view, and only its depth map is predicted. Feed-forward reconstruction models jointly predict geometry for all views, but also have to estimate the cameras, which leads to geometry distortion caused by reconstruction ambiguity. S2S-MVS bridges the two: it adopts a sequence-to-sequence formulation that predicts depth and point maps for all input views at once, while conditioning on known cameras through a single global cost volume.

Framework Overview

For each input view, we extract view tokens with a DINOv2 backbone and incorporate two camera-induced priors: Ray-map embeddings transform camera intrinsics and extrinsics into ray maps that are tokenized and merged with the image tokens, making the transformer camera-aware. Global-cost-volume embeddings replace the conventional per-view cost volumes with a unified cost volume that captures the 3D structure of the scene across all views. The tokens are then processed by attention layers that aggregate frame-level and global information, and decoding heads produce the depth maps and point maps of every input view.

Quantitative comparison on the RMVDB benchmark

We compare S2S-MVS with sequence-to-sequence feed-forward models (with and without ground-truth cameras, marked -w/c) and sequence-to-one MVS methods. The best, second-best and third-best results are highlighted. S2S-MVS obtains the highest AUC on ETH3D, KITTI, DTU and Tanks&Temples, and is considerably faster than the MVS baselines MVSFormer++ and MVSA.

Qualitative comparison of point map prediction

Qualitative comparison against DepthAnything3.

Predicted point maps are overlaid with the gray-coded ground truth to highlight alignment accuracy. DepthAnything3, with or without ground-truth cameras, suffers from 3D anisotropic scale distortion (e.g. on the Tanks&Temples scene in the last row), while S2S-MVS does not.

Global alignment compared to sequence-to-one MVS

Comparison of global alignment against MVSA.

Sequence-to-one methods like MVSA reconstruct each view independently, which can lead to misalignment across views. By reasoning about all views jointly, S2S-MVS produces globally consistent reconstructions.

Dense video reconstruction combined with SfM

Dense video-based reconstruction results.

Using camera parameters estimated by VidMap from video frames, S2S-MVS produces accurate dense reconstructions, whereas DepthAnything3 creates misaligned point clouds and camera offsets.

BibTeX

@inproceedings{fan2026s2smvs,
      author    = {Fan, Aoxiang and Dumery, Corentin and Talabot, Nicolas and Fua, Pascal},
      title     = {Revisiting Multi-View Stereo: A Sequence-to-Sequence Formulation},
      booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
      year      = {2026}
}