A score-based filter for nonlinear data assimilation

We propose a score-based generative sampling method for solving the nonlinear filtering problem with superior accuracy. A major drawback of existing nonlinear filtering methods, e.g., particle filters, is the low accuracy in handling high-dimensional nonlinear problems. To overcome this issue, we incorporate the score-based diffusion model into the recursive Bayesian filter framework to develop a novel score-based filter (SF). The key idea of SF is to store the information of the recursively updated filtering density function in the score function, instead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). By leveraging the reverse-time diffusion process, SF can generate unlimited samples to characterize the filtering density. An essential aspect of SF is its analytical update step, gradually incorporating data information into the score function. This step is crucial in mitigating the degeneracy issue faced when dealing with very high-dimensional nonlinear filtering problems. Three benchmark problems are used to demonstrate the performance of our method. In particular, SF provides surprisingly impressive performance in reliably capturing/tracking the 100-dimensional stochastic Lorenz system that is a well-known challenging problem for existing filtering methods.