Ensemble Machine Learning

Ensemble Learning is the combining diverse set of learners (individual models with low learning power) together to improv the stability and predictive power of the model. The learning algorithms which output only a single hypothesis tend to suffer from basically three issues. These issues are the statistical problem, the computational problem and the representation problem which can be partly overcome by applying ensemble methods. There are several ensemble learning models are random Forest, GBM, XGBoost , CatBoost etc.

• The learning algorithm which suffers from the statistical problem is said to have high variance in the input Data.
• The algorithm which exhibits the computational problem is sometimes described as having computational variance.
• The learning algorithm which suffers from the representational problem is said to have a high bias.

These three fundamental issues can be said as the three important ways in which existing linear learning algorithms like Logistic regression, Naïve Bayes fail.

Types of Ensemble Models

Some Commonly used ensembled techniques are bagging, boosting, stacking and blending which promises of reducing both the bias and the variance.

• Bagging: Bagging tries to implement similar learners on small sample populations and then takes an average or weighed average/voting or weighted voting of all the predictions. In generalized bagging, you can use different learners on different population.  As you can expect this helps us to reduce the variance error.
• Boosting: Boosting is an iterative technique which adjust the weight of an observation based on the last classification. If an observation was classified incorrectly, it tries to increase the weight of this observation and vice versa. Boosting in general decreases the bias error and builds strong predictive models. However, they may sometimes over fit on the training data.
• Stacking: This is a very interesting way of combining models. Here we use a learner to combine output from different learners. This can lead to decrease in either bias or variance error depending on the combining learner we use.