Decision Trees in Machine Learning

Decision tree is a predictive model used to create a workable model that will predict the value of a target variables. Decision tree is a type of supervised learning algorithm (having a pre-defined target variable) that is mostly used in classification problems. It works for both categorical and continuous input and output variables. In this technique, we split the population or sample into two or more homogeneous sets (or sub-populations) based on most significant splitter / differentiator in input variables.

Tree models where the target variable can take a finite set of values are called classification trees; in these tree structures, leaves represent class labels and branches represent conjunctionsof features that lead to those class labels. Decision trees where the target variable can take continuous values (real numbers) are called regression trees.

Many data mining software packages provide implementations of one or more decision tree algorithms. Several examples include  IBM-SPSS Modeler, RapidMiner, SAS Enterprise Miner, Matlab, R, Weka (open-source data mining suite), Orange, KNIME, & scikit-learn .

Uses of decision tree

• Financial institutions - One of the fundamental use cases is in option pricing, where a binary-like decision tree is used to predict the price of anoption in either bull or bear market.
• Marketers - To establish customers by type and predict whether a customer will buy a specfic type  of product.
• Medical Field - Models have been desiged to diagnose blood infections or even predict heart attack outcomes in chest pain patients. Variables in the decision tree include diagnosis, treatment, and patient data.
• Gaming Industry - It uses multiple decision trees in movement recognition and facial recognitiopn. 

Advantages of decision tree

• Simple to understand and interpret.

• Requires little data preparation.

• Able to handle both numerical and categorical data.

• Uses a white box model means the internal workings can be observed but not changed; you can view the steps that are being used when the tree is being modeled on the other hand in a black box model, the explanation for the results is typically difficult to understand, for example with an artificial neural network.

Limitations of Decision tree

• Decision-tree learners can create over-complex trees that do not generalise well from the training data.

• There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems.

• Practical decision-tree learning algorithms are based on heuristics such as the greedy algorithm where locally-optimal decisions are made at each node. Such algorithms cannot guarantee to return the globally-optimal decision tree. To reduce the greedy effect of local-optimality some methods such as the dual information distance (DID) tree were proposed.

• For data including categorical variables with different numbers of levels, information gain in decision trees is biased in favor of those attributes with more levels. However, the issue of biased predictor selection is avoided by the Conditional Inference approach.

Different types

Decision trees used in data mining are of two main types:

• Classification tree analysis is when the predicted outcome is the class to which the data belongs.

• Regression tree analysis is when the predicted outcome can be considered a real number (e.g. the price of a house, or a patient’s length of stay in a hospital).

The term Classification And Regression Tree (CART) analysis is an umbrella term used to refer to both of the above procedures.The procedure used to determine where to split the tree.

Some techniques, often called ensemble methods, construct more than one decision tree:

• Bagging decision trees, an early ensemble method, builds multiple decision trees by repeatedly resampling training data with replacement, and voting the trees for a consensus prediction.
• A Random Forest classifier uses a number of decision trees in order to improve the classification rate.
• Boosted Trees can be used for regression-type and classification-type problems.
• Rotation forest - in which every decision tree is trained by first applying principal component analysis (PCA) on a random subset of the input features.

Decision-tree algorithms:

• ID3 (Iterative Dichotomiser 3)
• C4.5 (successor of ID3)
• CART (Classification And Regression Tree)
• CHAID (CHi-squared Automatic Interaction Detector). Performs multi-level splits when computing classification trees.^[11]
• MARS: extends decision trees to handle numerical data better.
• Conditional Inference Trees.