Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations

Denis M. Akola, David F. Fouhey
New York University
Paper arXiv Code

Z3D takes source views and target camera pose and predicts the depth of the scene at the target camera position.

Abstract

3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-forward transformers. The scene representations learned by these models enable strong performance across multiple 3D vision tasks.

In this paper, we investigate using their internal representations to infer 3D structure in a scene from novel viewpoints. Our hypothesis is that, in order to solve the task of 3D reconstruction, these models must learn representations that capture a substantial amount of general knowledge about 3D scenes.

After demonstrating that hidden surfaces can be decoded from the internal representations of a 3DFM, we propose Z3D, a method that estimates pointmaps in unseen views by performing latent diffusion directly on 3DFM representations. We show that Z3D predicts realistic depth maps for novel views across multiple datasets, demonstrating that pretrained 3D foundation models provide a powerful latent space for generative 3D reconstruction.

Video

1 Source view - 1 Target view : Depth Map Comparison

Source Image source
GT gt depth
Target Image target
VGGT vggt depth
VGGT-DD dd depth
VGGT-Z3D z3d depth
WM wm depth
WM-DD wm-dd depth
WM-Z3D wm-z3d depth

From the visualizations above, we observe that depth maps predicted by the depth-diffusion baseline (VGGT-DD, WM-DD) contain patchy artifacts, whereas Z3D counterparts (VGGT-Z3D, WM-Z3D) yields smoother, more coherent depth structure. NB: Both Z3D and DD models do not recieve the target view image(s) as input(s).

Pick an example above to switch scenes.

Depth Gradient Comparison - VGGT

Source Image source
Target Image target
GT gt depth
VGGT vggt depth
VGGT-DD vggt-dd depth
VGGT-Z3D z3d-vggt depth
VGGT-DD Depth Grad vggt-dd depth gradient
VGGT-Z3D Depth Grad z3d-vggt depth gradient
VGGT-DD vggt-dd zoomed depth gradient
VGGT-Z3D z3d-vggt zoomed depth gradient

From the visualizations above, we observe that depth maps predicted by the depth-diffusion baseline (VGGT-DD) contain patchy artifacts. The corresponding depth gradients further reveal that VGGT-DD produces noisy, inconsistent gradients, whereas VGGT-Z3D yields smoother, more coherent depth structure. The last column shows the zoomed in view of the highlighted section in the depth gradient images.

Pick an example above to switch scenes.

Depth Gradient Comparison - World Mirror

Source Image source
Target Image target
GT gt depth
WM wm depth
WM-DD wm-dd depth
WM-Z3D z3d-wm depth
WM-DD Depth Grad wm-dd depth gradient
WM-Z3D Depth Grad z3d-wm depth gradient
WM-DD wm-dd zoomed depth gradient
WM-Z3D wm-z3d zoomed depth gradient

Pick an example above to switch scenes.

From the visualizations above, we observe that depth maps predicted by the depth-diffusion baseline (WM-DD) contain patchy artifacts. The corresponding depth gradients further reveal that WM-DD produces noisy, inconsistent gradients, whereas WM-Z3D yields smoother, more coherent depth structure. The last column shows the zoomed in view of the highlighted section in the depth gradient images.

1 Source - 1 Target Point Cloud Comparison

Source Image source
GT
Target Image target
VGGT-DD
VGGT
VGGT-Z3D
WM
WM-DD
WM-Z3D

From the visualizations above, we observe that point clouds obtained by projecting the depth-diffusion baselines' predicted depth maps (WM-DD and VGGT-DD) are considerably noisier, with scattered, uneven surfaces. Their Z3D counterparts, by contrast, produce smooth, coherent point clouds.

Drag to rotate — all point clouds stay in sync.

See more examples →

Multi-Targets (2 Source views, 4 target views) Point Cloud Visualization

Source Views

Source View 1 source view 1
Source View 2 source view 2

Target Views (4 target viewpoints)

Target View 1target view 1
Target View 2target view 2
Target View 3target view 3
Target View 4target view 4

Combined Point Clouds (all target views fused)

GT
VGGT-Z3D
WM-Z3D

From the visualizations above, we observe that point clouds obtained by projecting Z3D's predicted depth maps are smooth and coherent.

Drag to rotate — all three point clouds stay in sync.

See more examples →

BibTeX

@inproceedings{akola2026z3d,
  author    = {Akola, Denis and David, Fouhey},
  title     = {Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations},
  journal   = {ECCV},
  year      = {2026},
}