Unlocking Model Evaluation: How the CRITIC Method Calculates Criterion Weights with Python

CRITIC Method

CRITIC (Criteria Importance Through Intercriteria Correlation) is a technique for evaluating models by analysing the correlation among multiple performance metrics to determine each metric’s importance.

The method follows these steps:

Identify evaluation criteria (e.g., accuracy, recall, F1).

Evaluate the model using the selected criteria and record scores.

Compute correlation coefficients between each pair of criteria.

Determine criterion importance using the correlation matrix.

Analyze results to highlight model strengths and weaknesses and guide improvements.

Mathematical Model

The CRITIC method assumes a set of feasible alternatives (i = 1,…,m) and evaluation criteria.

Step 1: Construct the data matrix representing each alternative’s performance on each criterion.

Let x_{ij} denote the measurement of criterion j for alternative i.

Step 2: Normalize the data using range normalization.

For benefit criteria:

For cost criteria:

Step 3: Determine weights.

When calculating standard weights, both the standard deviation of each criterion and its correlation with other criteria are considered. The weight for criterion j is calculated using its standard deviation σ_j and the correlation coefficients r_{jk}; larger values indicate greater information content and higher relative importance.

Case Study and Python Implementation

Consider four decision alternatives (A1–A4) and four attribute criteria. The CRITIC method is implemented in Python to compute the criterion weights:

<code>import numpy as np
import pandas as pd

data = {'A1': [7, 6, 5, 7],
        'A2': [9, 7, 6, 8],
        'A3': [6, 5, 8, 6],
        'A4': [8, 7, 7, 9]}

df = pd.DataFrame(data)
# Build data matrix
X = df.values

# Range normalization
X_norm = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))

# Determine weights
sigma = np.std(X_norm, axis=0)
corr = np.corrcoef(X_norm.T)
C = sigma * np.sum(1 - corr, axis=0)
weights = C / np.sum(C)

print('权重:', weights)
</code>

The resulting weights are [0.20480263 0.1875107 0.4188841 0.18880257] .

References:

HYBRID APPROACH CRITIC‑TOPSIS FOR CLOUD SERVICE SELECTION

Adalı, Esra Aytaç. "CRITIC and MAUT methods for the contract manufacturer selection problem." European Journal of Multidisciplinary Studies 2.5 (2017): 93‑101.