Decision Trees and Boosting, XGBoost | Two Minute Papers #55

Decision Trees and Boosting, XGBoost | Two Minute Papers #55


Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. A decision tree is a great tool to help making
good decisions from a huge bunch of data. The classical example is when we have a bunch
of information about people and would like to find out whether they like computer games
or not. Note that this is a toy example for educational purposes. We can build the following tree: if the person’s
age in question is over 15, the person is less likely to like computer games. If the
subject is under 15 and is a male, he is quite likely to like video games, if she’s female,
then less likely. Note that the output of the tree can be a
decision, like yes or no, but in our case, we will assign positive and negative scores
instead. You’ll see in a minute why that’s beneficial. But this tree wa s just one possible way of
approaching the problem, and admittedly, not a spectacular one – a different decision tree
could be simply asking whether this person uses a computer daily or not. Individually, these trees are quite shallow
and we call them weak learners. This term means that individually, they are quite inaccurate,
but slightly better than random guessing. And now comes the cool part. The concept of
tree boosting means that we take many weak learners and combine them into a strong learner.
Using the mentioned scoring system instead of decisions also makes this process easy
and straightforward to implement. Boosting is similar to what we do with illnesses.
If a doctor says that I have a rare condition, I will make sure and ask at least a few more
doctors to make a more educated decision about my health. The cool thing is that the individual trees
don’t have to be great, if they give you decisions that are just a bit better than random guessing,
using a lot of them will produce strong learning results. If we go back to the analogy with doctors,
then if the individual doctors know just enough not to kill the patient, a well-chosen committee
will be able to put together an accurate diagnosis for the patient. An even cooler, adaptive version of this technique
brings in new doctors to the committee according to the deficiencies of the existing members. One other huge advantage of boosted trees
over neural networks is that we actually see why and how the computer arrives to a decision. This is a remarkably simple method that leads
to results of very respectable accuracy. A well-known software library called XGBoost
has been responsible for winning a staggering amount of machine learning competitions in
Kaggle. I’d like to take a second to thank you Fellow
Scholars for your amazing support on Patreon and making Two Minute Papers possible. Creating
these episodes is a lot of hard work and your support has been invaluable so far, thank
you so much! We used to have three categories for supporters.
Undergrad students get access to a Patron-only activity feed and get to know well in advance
the topics of the new episodes. PhD students who are addicted to Two Minute Papers get
a chance to see every episode up to 24 hours in advance. Talking about committees in this
episode, Full Professors form a Committee to decide the order of the next few episodes. And now, we introduce a new category, the
Nobel Laureate. Supporters in this category can literally become a part of Two Minute
Papers and will be listed in the video description box in the upcoming episodes. Plus all of
the above. Thanks for watching, and for your generous
support, and I’ll see you next time!

12 thoughts on “Decision Trees and Boosting, XGBoost | Two Minute Papers #55

Leave a Reply

Leave a Reply

Your email address will not be published. Required fields are marked *