Ensemble learning is being considered as one of the most well-established and efficient techniques in the contemporary machine learning. The key to the satisfactory performance of such combined models lies in the supplied base learners and selected combination strategy. In this paper we will focus on the former issue. Having classifiers that are of high individual quality and complementary to each other is a desirable property. Among several ways to ensure diversity feature space division deserves attention. The most popular method employed here is Random Subspace approach. However, due to its random nature one cannot consider this approach as stable one or suitable for real-life applications. Therefore, we propose a new approach called Deterministic Subspace that constructs feature subspaces in a guided and repetitive manner. We present a general framework and three dedicated measures that can be used for selecting diverse and uncorrelated features for each base learner. This way we will always obtain identical sets of features, leading to creation of stable ensembles. Experimental study backed-up with statistical analysis prove the usefulness of our method in comparison to popular randomized solution.
Forming Classifier Ensembles with Deterministic Feature Subspaces. In Proceedings of the 2016 Federated Conference on Computer Science and Information Systems.
- GitHub page (full code, sufficient to repeat experiment)
- Complete results in CSV format
- Visualization of results for specific datasets and base learners. Classification accuracy used as feature quality measure.
Support Vector Machine
Parzen Kernel Density Estimation
Nearest Neighbour Kernel Density Estimation
Gaussian Mixture Model