Master Ensemble Learning: Bagging, Boosting, and Stacking with Python

Ensemble Learning

In machine learning we aim to improve model accuracy, and one effective way is to combine multiple weak models—a technique known as ensemble learning. By aggregating several models, the final prediction becomes more precise.

Three Types of Ensemble Models

The most common ensemble approaches are:

Bagging (Bootstrap Aggregating): train many independent models in parallel and aggregate their outputs (e.g., majority voting for classification, averaging for regression).

Boosting: train models sequentially, where each new model tries to correct the errors of the previous one.

Stacking: train diverse models in parallel and then use a meta‑model to combine their predictions.

Bagging Method

Using a decision‑tree as the base estimator, we train 100 trees on the classic load_iris dataset (features: sepal length, sepal width, petal length, petal width; target classes: 0, 1, 2). The core Bagging code is:

<code>from sklearn.ensemble import BaggingClassifier
model = BaggingClassifier(base_estimator=cart, n_estimators=num_trees, random_state=seed)</code>

Full implementation:

<code># import libraries
import numpy as np
from sklearn import model_selection
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

# load data
iris = load_iris()
X = iris.data
Y = iris.target

# split 70/30
X_fit, X_eval, y_fit, y_test = model_selection.train_test_split(X, Y, test_size=0.30, random_state=1)

# 10‑fold cross‑validation
kfold = model_selection.KFold(n_splits=10)

# base decision tree
cart = DecisionTreeClassifier()
num_trees = 100
seed = 121
model = BaggingClassifier(base_estimator=cart, n_estimators=num_trees, random_state=seed)

# cross‑validation scores
results = model_selection.cross_val_score(model, X_fit, y_fit, cv=kfold)
for i in range(len(results)):
    print("Model: " + str(i) + " Accuracy is: " + str(results[i]))
print("Mean Accuracy is: " + str(results.mean()))

# train and evaluate
model.fit(X_fit, y_fit)
pred_label = model.predict(X_eval)
nnz = np.shape(y_test)[0] - np.count_nonzero(pred_label - y_test)
acc = 100 * nnz / np.shape(y_test)[0]
print('accuracy is: ' + str(acc))
</code>

Running this code yields individual fold accuracies (most are 1.0) and a mean accuracy of about 95.09%, corresponding to an overall accuracy of 95.56% on the test set.

Using the same idea, the bagging process can be replaced by a ready‑made Random Forest:

<code>from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=num_trees)
model.fit(X_fit, y_fit)
pred_label = model.predict(X_eval)
</code>

We can also swap the base estimator for a Support Vector Machine:

<code>from sklearn.svm import SVC
model = SVC()
model.fit(X_fit, y_fit)
pred_label = model.predict(X_eval)
# compute accuracy as before
</code>

Accuracy with SVM reaches 97.77% on the same split.

Boosting

Boosting combines weak learners sequentially. The example uses AdaBoost with decision trees as base learners:

<code>from sklearn.ensemble import AdaBoostClassifier
cart = DecisionTreeClassifier()
num_trees = 100
model = AdaBoostClassifier(base_estimator=cart, n_estimators=num_trees, learning_rate=0.1)
model.fit(X_fit, y_fit)
# compute accuracy as before
</code>

The resulting accuracy is around 97%.

Stacking

Stacking blends predictions from heterogeneous models using a meta‑learner. The workflow trains Decision Tree, K‑Nearest Neighbors, Support Vector Machine, Gaussian Naïve Bayes, and Logistic Regression as level‑0 models, then feeds their predictions into a Logistic Regression meta‑model:

<code>from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import StackingClassifier

level0 = []
level0.append(('lr', LogisticRegression()))
level0.append(('knn', KNeighborsClassifier()))
level0.append(('cart', DecisionTreeClassifier()))
level0.append(('svm', SVC()))
level0.append(('bayes', GaussianNB()))
level1 = LogisticRegression()
model = StackingClassifier(estimators=level0, final_estimator=level1)
model.fit(X_fit, y_fit)
# compute accuracy as before
</code>

Stacking also achieves about 97.77% accuracy on the Iris dataset. The similar performance across Bagging, Boosting, and Stacking is mainly due to the simplicity of the Iris data; different datasets may show larger differences.

Overall, ensemble strategies provide practical ways to boost model accuracy.