Skip to main content

Decision Trees Introduction

Decision Trees are one of the most popular and intuitive machine learning algorithms used for both classification and regression tasks. They work by breaking down a dataset into smaller subsets while at the same time incrementally developing an associated decision tree. The final tree structure consists of decision nodes and leaf nodes, providing a flowchart-like model that makes decisions based on the input features.

In this article, we will explore:

  • What Decision Trees are and how they work.
  • Why they are widely used in machine learning.
  • Their strengths and limitations.

1. What are Decision Trees?​

At their core, Decision Trees are hierarchical models that recursively split the data into subsets based on input-feature values. Different algorithms support different split types. scikit-learn's CART implementation builds binary threshold splits over numeric input columns; it does not currently consume categorical columns directly. The structure resembles an inverted tree where:

  • Root Node: This represents the entire dataset and the first feature split.
  • Decision Nodes: These nodes represent a feature on which the data is split.
  • Leaf Nodes: These nodes represent the final outcome (class label or value).

Example of a Decision Tree:​

Below is a simple diagram of how a Decision Tree might look for classifying whether to play tennis based on weather conditions:

                [Outlook]
               /    |    \
         Sunny   Overcast   Rain
        /     \              |
   Humidity   Play          Windy
   /     \                   /   \
High   Normal               Yes    No

The tree makes decisions at each node by asking questions based on the input features (e.g., "Is the outlook sunny?"). Based on the answer, it follows a branch down to the next decision or leaf node.

Key Concepts:​

  • Splitting: The process of dividing a node into two or more sub-nodes based on a feature value.
  • Impurity: A measure of how mixed the classes are in a given node. The goal of splitting is to reduce impurity.
  • Gini Index and Entropy: These are common metrics used to evaluate how well a node splits the data.

2. How Decision Trees Work​

Decision Trees operate by asking a series of if-else questions about the features of the dataset. For each node in the tree, the algorithm chooses a feature that best splits the data into two or more parts based on some criterion (such as Gini Impurity or Information Gain).

Steps to Build a Decision Tree:​

  1. Select the Best Feature to Split On:

    • The algorithm evaluates all possible features and selects the one that best separates the data. It uses metrics like Gini Impurity or Information Gain to quantify the "goodness" of a split.

  2. Recursive Splitting:

    • The process repeats for each subset of the data. Each time, a new feature is chosen to further split the subset until the data is pure (i.e., only contains one class) or a stopping criterion is met.

  3. Leaf Nodes:

    • Once the data can no longer be split or a stopping criterion is reached (e.g., maximum depth, minimum samples per leaf), the algorithm assigns a label to the leaf node (for classification) or a value (for regression).

Splitting Criteria:​

  1. Gini Impurity:

    • Measures the "impurity" of a node by calculating the probability of a random sample being misclassified if it was labeled according to the class distribution at the node.
    • Formula:
    Gini=1−∑i=1kpi2\text{Gini} = 1 - \sum_{i=1}^{k} p_i^2

    Where pip_i is the probability of class ii at the node.

  2. Information Gain (Entropy):

    • Measures how much information is gained by splitting the data on a given feature.
    • Formula:
    Entropy=−∑i=1kpilog⁡2(pi)\text{Entropy} = - \sum_{i=1}^{k} p_i \log_2(p_i)

    Information Gain is the reduction in entropy after a split.

3. Why Use Decision Trees?​

Decision Trees are favored in machine learning for several reasons:

3.1. Interpretability​

  • Decision Trees are easy to visualize and interpret. The path from the root node to a leaf provides a clear sequence of decisions that lead to a classification or prediction.

3.2. Versatility​

  • Decision Trees can be applied to both classification and regression tasks. Some tree algorithms handle categorical predictors directly, but scikit-learn's implementation requires a numeric representation. Encoding choices can change which partitions the tree can learn.

3.3. No Assumptions About Data Distribution​

  • Unlike many other machine learning models, Decision Trees do not require the data to follow any specific distribution (e.g., linear relationships or normal distribution).

3.4. Non-Linear Decision Boundaries​

  • Decision Trees can easily capture non-linear relationships between features and the target variable.

3.5. Limited Preprocessing, with Explicit Contracts​

  • Threshold trees do not require standardization. Missing-value behavior is implementation-specific: current scikit-learn trees support missing values only for documented splitter/criterion combinations and learn a routing direction for them; other libraries may reject them, use surrogate splits, or treat missingness as a category. Verify the chosen estimator rather than assuming a universal policy.

4. Limitations of Decision Trees​

While Decision Trees offer many advantages, they also come with several limitations:

4.1. Overfitting​

  • Decision Trees are prone to overfitting, especially when the tree grows too deep. This can result in poor generalization to new, unseen data.

Solution: Pruning the tree, setting a maximum depth, or using ensemble methods like Random Forests can help combat overfitting.

4.2. Instability​

  • Small changes in the data can lead to completely different tree structures, making Decision Trees unstable.

Solution: Using techniques like Random Forests or Gradient Boosting can help stabilize model predictions.

4.3. Greedy Splits and Biased Importance​

  • Greedy search may choose a locally useful split that prevents a better later partition. Impurity-based feature importance can also favor continuous or high-cardinality predictors and divide credit unpredictably among correlated features.

Solution: Validate predictive performance on untouched data, regularize the tree, and treat impurity importance as a property of the fitted model—not a causal ranking. Compare held-out permutation importance when predictive reliance matters.

5. Conclusion​

Decision Trees are a highly intuitive and flexible machine learning algorithm. Their ability to handle both classification and regression, along with their transparency and ease of use, make them a popular choice across many fields. However, they can be prone to overfitting and instability, which can be addressed through proper regularization and ensemble techniques.

In the next sections, we will examine impurity and pruning, train a verified scikit-learn tree, and clarify where TensorFlow and PyTorch do—and do not—provide maintained classical tree estimators.