Recovery of Herschel-Bulkley fluid parameters from video via differentiable rendering and simulations

Recreating physical behavior of fluids in simulations based on real-world footage remains a challenge, especially for non-Newtonian fluids. This work introduces a novel method combining Neural Radiance Fields (NeRFs), which use deep neural networks to map 3D scene coordinates to color and density, with the Material Point Method (MPM), a simulation technique representing materials by moving points capable of large deformation. We seek to accurately recover physical parameters and achieve high-fidelity 3D reconstructions of a Herschel-Bulkley fluid, ketchup, from a single real-world video, by leveraging the differentiable nature of NeRF and the fluid simulation capabilities of MPM. Key results include the retrieval of constitutive parameters, such as modulus, yield stress, and viscosity, as well as reconstructed videos that reflect the behavior of the training video. Thanks to the recovered material parameters, we in turn achieve predictive simulations of the fluid. Overall, our method provides the framework for video-based property extraction.