XGBoost: A Scalable Tree Boosting System

XGBoost asks how many small decision trees can correct one another without growing into an uncontrolled forest, and how that learning rule can survive sparse, billion-row data. Read it to connect additive leaf scores and regularized split gain to weighted candidate sketches, learned missing-value directions, cache-aware column blocks, and the system work behind a practical tabular-ML classic.

Reading focus: How each new tree contributes a signed leaf correction to the ensemble's current prediction instead of replacing what earlier trees learned. Why a split survives only when its improvement clears a complexity cost, and how shrinkage and leaf-weight penalties restrain the corrections. How weighted split candidates, learned default directions for missing values, sorted column blocks, prefetching, compression, and sharding turn the same learner into a scalable system.

KDD 2016. Chen and Guestrin. 35 min read, easy difficulty.