A Refined Margin Distribution Analysis for Forest Representation Learning

Abstract

In this paper, we formulate the forest representation learning approach called CasDF as an additive model which boosts the augmented feature instead of the prediction. We substantially improve the upper bound of the generalization gap from O( m/m) to O( m/m), while the margin ratio of the margin standard deviation to the margin mean is sufficiently small. This tighter upper bound inspires us to optimize the ratio. Therefore, we design a margin distribution reweighting approach for deep forest to achieve a small margin ratio by boosting the augmented feature. Experiments confirm the correlation between the margin distribution and generalization performance. We remark that this study offers a novel understanding of CasDF from the perspective of the margin theory and further guides the layer-by-layer forest representation learning.

Tasks

Reproductions