Sky-NeRF: Learning 4D Cloud Topography in a Dynamic Neural Radiance Field
XXV ISPRS Congress
Abstract
We present Sky-NeRF, a novel method for cloud topography estimation based on Dynamic Neural Radiance Fields. Similar to NeRF, we propose to model the 3D structure of clouds as a radiance field, encoded in the parameters of a neural representation. Our goal is to reconstruct the 3D geometry, appearance, and motion of the cloud using a stereo-video of high-resolution top of the atmosphere radiance images. In this paper, we evaluate a novel way of modeling the dynamic behavior of clouds, with the goal of extracting added-value physical information regarding the cloud such as advection speed and direction, velocity field and cloud trajectories. We investigate how to include a simple physical prior, advection, into the learning system and evaluate its impact. Our results show that Sky-NeRF is able to provide a more complete 4D reconstruction than traditional stereo-matching-based algorithms. Moreover, thanks to a physics-based interpolation, Sky-NeRF is able to generate coherent new images from unseen viewing angles, and at any time between the observed frames.
Method overview
Geometrical configuration of the dataset and overview of Sky-NeRF. Drawing inspiration from D-NeRF, At any time \(t\), 3D samples \(X\) are collected by ray casting (a). Each sample is passed to an advection+residual decomposition \(\psi^{\to C}_t = \boldsymbol{u}_{adv} + \hat{\psi}_t^{\to C}\) which maps position \(\boldsymbol{X}\) in ''local'' space at any time \(t\) to position \(\boldsymbol{X}_C\) in a time-independent ''canonical space'' (b), where the 3D opacity and color distributions are encoded by a network \(\psi_x\) (c). The differentiable renderer then blends the predicted samples along the ray to produce a pixel colour (d), compared with the true color through \(\mathcal{L}_{RGB}\) (e). For trajectory estimation, the position \(X'\) of any point \(X_C\) of the canonical space at any time \(t'\) is estimated by \(\psi_t^{\to L}\) which encodes the reverse mapping of \(\psi_t^{\to C}\) (f), supervised through the consistency loss term \(\mathcal{L}_{\to{L}}\) when \(t'=t\) (g).
3D reconstruction
Input sequence. The input sequence consists of red-band optical images simulated to match the geometrical configuration of the C3IEL mission. The views match the positions of two satellite cameras that follow the same fly-by pass over the cloud scene with a baseline of 150km between them. The images are simulated at a rate of one pair every 20s over 180s. More detailed information regarding the simulation is provided in the referenced publications [1], [2], [3].
3D reconstruction. Sky-NeRF can reconstruct the cloud scene's geometry at any time during the satellite pass. The above clouds are reconstructed 100s after the start of the sequence. Although stereo-vision pipeline CARS (right) keeps a slight advantage in reconstruction accuracy over the cloud top, Sky-NeRF (middle) reconstructs a more complete geometry when compared with the reference produced by the simulation (left).
Video synthesis
Sky-NeRF can generate intuitive visualizations of the clouds and their development. The videos below were produced by fixing the camera position and unfolding time, then by fixing time and executing a fly-by. As interpolating further from the satellites'positions at a given time makes for a harder problem, in the fixed-position case, reconstruction quality tends to decay towards the temporal extremities of the sequence.
Unfolding time at fixed, nadir position.
Fly-by visualization, 90s after the start of the sequence.
Motion estimation
Thanks to its advection module and canonical-to-local mapping network, Sky-NeRF is able to approximate the trajectory field of the cloud between any two points in time, while separating a global-scene advection motion from residual cloud motions.
Example trajectory predicted for a surface point on a cloud of the scene, from acquisition \(A_1\) to \(A_{19}\). Slices of the reference point cloud are shown at three acquisitions. The predicted trajectory is compared with a reference trajectory computed by concatenating Iterative Closest Point transformations between successive acquisitions.
On the left, the magnitude of the predicted advection, overlaid to the velocities derived between successive acquisition times from points of the reference point clouds. The vertical profile of the learnt wind is consistent with the average motion of the reference. On the right, the elevation and azimuth of the advection are estimated accurately.
BibTeX
@article{terrisse2026skynerf,
title={Sky-NeRF: Learning 4D Cloud Topography in a Dynamic Neural Radiance Field},
author={Theïlo Terrisse and Dawa Derksen and David Youssefi and Hugo Meric},
journal={ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
volume={XI-2-2026},
year={2026},
pages={535--544},
DOI = {10.5194/isprs-annals-XI-2-2026-535-2026}
}Acknowledgments
We gratefully acknowledge the Laboratoire d'Optique Atmosphérique for providing access to the cloud simulation used in this study. We also sincerely thank CNES for supporting this work by granting access to high-performance computing resources at their computing center.
References
[1] Dandini, P., Cornet, C., Binet, R., Fenouil, L., Holodovsky, V., Y. Schechner, Y., Ricard, D., Rosenfeld, D., 2022. 3D cloud envelope and cloud development velocity from simulated CLOUD (C3IEL) stereo images. Atmospheric Measurement Techniques, 15(20), 6221-6242.
[2] Cornet, C., C-Labonnote, L., Szczap, F., 2010. Three dimensional polarized Monte Carlo atmospheric radiative transfer model (3DMCPOL): 3D effects on polarized visible reflectances of a cirrus cloud. Journal of Quantitative Spectroscopy and Radiative Transfer, 111(1), 174-186.
[3] Strauss, C., Ricard, D., Lac, C., Verrelle, A., 2019. Evaluation of turbulence parametrizations in convective clouds and their environment based on a large-eddy simulation. Quarterly Journal of the Royal Meteorological Society, 145(724), 3195-3217.