Modern 3D reconstruction and novel view synthesis approaches have demonstrated strong performance on scenes with opaque, non-refractive objects. However, most assume straight light paths and therefore cannot properly handle refractive and reflective materials. The lack of datasets specialized for these effects has impeded efforts to fairly and thoroughly evaluate performance and thereby make progress in this domain. Most existing datasets focus on opaque scenes, while those targeting refractive objects are often limited to thin glass with negligible ray bending, untinted single-material objects, or backgrounds treated as infinitely distant, leaving key failure modes under strong refraction untested. To address this gap, we introduce the RefRef dataset. It contains 150 synthetic and 60 real scenes spanning varying levels of geometric complexity and diverse background types that expose the limitations of existing methods. For the purpose of benchmarking, we also provide an oracle method that, given the object geometry and refractive indices, calculates accurate light paths for neural rendering, and a simple two-stage baseline that relaxes these assumptions. We evaluate these against state-of-the-art methods and show that the task is far from solved.
Unlike existing datasets, RefRef contains objects with large refractive volumes, both clear and tinted materials, single and multi-material objects, and backgrounds from nearby geometry to distant environments. Across the scenes these appear in different combinations, producing strong light bending, multiple refractions, total internal reflection and light absorption.
| Category | Synthetic | Real |
|---|---|---|
| Convex single-material e.g., glass sphere, glass pyramid | 27 | 8 |
| Nonconvex single-material e.g., glass sculptures, jars, magnifiers | 60 | 4 |
| Nonconvex multi-material e.g., reed diffusers, wine bottles, snow globes | 63 | 48 |
Given the ground-truth object geometry and refractive indices, the oracle traces physically accurate light paths. Samples along each camera ray are updated at every surface intersection, and refracted and reflected rays are handled separately. A Zip-NeRF field predicts color and density for each sample, and the rendered colors are combined with the Fresnel equations. The oracle models refraction, reflection, total internal reflection and absorption.
R3F (Refractive–Reflective Radiance Field) removes the need for ground-truth geometry and refractive indices. It first estimates the object's geometry and refractive index with NU-NeRF, then fits a Zip-NeRF background field with the object masked out (masks predicted by SAM 3). Finally, it fits a separate in-object field using the oracle's ray tracing while the background field stays frozen, which prevents floaters outside the object. Its performance depends entirely on the quality of the estimated geometry.
On this real scene with several objects, every method fails, including R3F.
Novel view synthesis (PSNR, masked PSNR, SSIM, LPIPS) and geometry (distance-map MAE) on the RefRef test sets. On synthetic scenes, the oracle is far ahead of every other method, including R3F. On real scenes, every method does poorly, and even the top scorer (Zip-NeRF) produces very blurry renderings.
Bold: best. Underline: second best. † The oracle has access to the ground-truth object geometry and material properties. A dash means the method could not produce that result. DMAE is not reported for real scenes, which have no ground-truth geometry.
@misc{yin2025refref,
title = {RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects},
author = {Yin, Yue and Tao, Enze and Deng, Weijian and Campbell, Dylan},
year = {2025},
eprint = {2505.05848},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2505.05848}
}