Nearest neighbor projective fuser for function estimation

There is currently a wide choice of function estimators, and it is often more effective and practical to fuse them rather than choosing a "best" one. An optimal projective fuser was proposed earlier based on the lower envelope of error regressions of the estimators. In most practical cases, however, the error regressions are not available and only a finite sample is given. Consequently this optimal fuser is hard to implement and furthermore guarantees only the asymptotic consistency. In this paper, we propose a projective fuser based on the nearest neighbor concept, which is easy to implement. Under fairly general smoothness and non-smoothness conditions on the individual estimators, we show that this fuser's expected error is close to optimal with a high probability, for a finite sample and irrespective of the underlying distributions. This performance guarantee is stronger than the previous ones for projective fusers and also implies asymptotic consistency. The required smoothness condition, namely Lipschitz continuity, is satisfied by sigmoid neural networks and certain radial-basis functions. The non-smoothness condition requires bounded variation which is satisfied by k-nearest neighbor, regressogram,regression tree, Nadaraya-Watson and feedforward threshold network estimators.