Understanding Common Loss Functions Across Machine Learning Models

In machine learning, the core objective of algorithms is to minimize or optimize a function known as the loss (or cost) function, which measures the discrepancy between predictions and actual values.

Linear Regression: Mean Squared Error (MSE)

Linear regression seeks the best‑fit line for the data, using the mean squared error (MSE) as its loss function.

Here, y is the actual value, ŷ is the predicted value, and n is the number of observations.

Logistic Regression: Cross‑Entropy Loss

Logistic regression addresses binary classification problems and employs cross‑entropy (log loss) as its loss function.

In this case, y denotes the true label (0 or 1) and p is the predicted probability of the positive class.

Decision Trees and Random Forests

Classifiers use Gini impurity or information gain.

Regressors use mean squared error (MSE).

Support Vector Machine (SVM): Hinge Loss

SVMs use hinge loss as their loss function.

Here, y is the true label (±1) and f(x) is the predicted value.

Neural Networks

Neural networks can tackle various problem types, thus they can use multiple loss functions.

Regression: mean squared error (MSE).

Classification: cross‑entropy loss.

AdaBoost: Exponential Loss

AdaBoost is an ensemble learning algorithm that combines weak classifiers into a strong one; at each iteration it assigns higher weights to mis‑classified instances and minimizes a weighted exponential loss.

In this context, y is the true label and p is the predicted value.