Gradient Boosting, Decision Trees and XGBoost with CUDA
Gradient boosting is a powerful machine learning algorithm used to achieve state-of-the-art accuracy on a variety of tasks such as regression, classification and ranking. It has achieved notice in machine learning competitions in recent years by “winning practically every competition in the structured data category”. If you don’t use deep neural networks for your problem, there is a good chance you use gradient boosting.
Rory Mitchell, PhD student at University of Waikato, implemented the multi-GPU gradient boosting algorithm in the XGBoost library (this https URL). This algorithm allows fast, scalable training on multi-GPU systems with all of the features of the XGBoost library. It employs data compression techniques to minimise the usage of scarce GPU memory while still allowing highly efficient implementation. Using this algorithm, it is possible to process 115 million training instances in under three minutes on a publicly available cloud computing instance. The algorithm is implemented using end-to-end GPU parallelism, with prediction, gradient calculation, feature quantisation, decision tree construction and evaluation phases all computed on device.
- Rory Mitchell
- Eibe Frank