An ensemble score filter for tracking high-dimensional nonlinear dynamical systems

We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF addresses this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterize the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, instead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation method that uses a mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial–temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems while also saving a tremendous amount of time spent on training neural networks. High-dimensional Lorenz-96 systems are used to demonstrate the performance of our method. EnSF provides superior performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter, in reliably and efficiently tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimensions) with highly nonlinear observation processes.