GBDT#

Suppose a regression problem, the objective function:

\[\mbox{obj} = \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i})\]

Besides fitting the residual directly like boosting tree, we can fit the negative gradient to descent.

Algorithm#

GBDT for regression:

  1. compute each instance’s gradient:

\[r_{mi} = -\left[\frac{\partial{L(y_{i}, f(x_{i}))}}{\partial f(x_{i})}\right]_{f = f_{m-1}}\]

  1. generate a tree that fits \(r_{mi}\), denote it’s leaf nodes area as \(R_{mj}, j=1,...,J\).

  2. minimize loss inside each \(R_{mj}\):

\[c_{mj} = \underset{c}{\mbox{argmin}}\sum_{x_{i}\in{R_{mj}}}L(y_{i}, f_{m-1}(x_{i}) + c)\]

  1. update \(f_{m}(x) = f_{m-1}(x) + \sum_{j=1}^{J}c_{mj}I(x \in R_{mj})\).

Examples#

import  numpy as np

np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(100)

from sklearn.ensemble import GradientBoostingRegressor

gbrt = GradientBoostingRegressor(max_depth=2, 
                                 n_estimators=3, 
                                 learning_rate=1.0, 
                                 random_state=42)
gbrt.fit(X, y)

GradientBoostingRegressor(learning_rate=1.0, max_depth=2, n_estimators=3,
                          random_state=42)