Comprehensive Guide to Feature Engineering and Data Preprocessing for Machine Learning

Feature engineering leverages domain knowledge and existing data to create new features that improve the performance of machine learning algorithms. It can be performed manually or automatically, and is essential for traditional machine learning and data mining tasks.

Feature Understanding distinguishes structured vs. unstructured data and quantitative vs. qualitative attributes, providing the foundation for subsequent processing.

Feature Cleaning addresses data alignment (time formats, field consistency, unit consistency), missing value handling (deletion, mean/median/mode imputation, model‑based prediction, interpolation, multiple imputation), and outlier detection (statistical methods, 3σ rule, box‑plot, distance‑based, density‑based, clustering‑based techniques). Example code for loading the Titanic dataset and checking missing values:

import pandas as pd
import seaborn as sns

df_titanic = sns.load_dataset('titanic')
print(df_titanic.isnull().sum())

Feature Construction includes statistical feature creation (e.g., quartiles, means), periodic features (e.g., rolling windows), binning methods (equal‑width, equal‑frequency, Chi‑Merge, chi‑square, entropy), and feature combinations (Cartesian products for categorical‑categorical, grouping for categorical‑continuous, arithmetic for continuous‑continuous). Sample code for age binning:

def age_bin(x):
    if x <= 18:
        return 'child'
    elif x <= 30:
        return 'young'
    elif x <= 55:
        return 'midlife'
    else:
        return 'old'

df_titanic['age_bin'] = df_titanic['age'].map(age_bin)

Feature Selection is divided into filter (variance threshold, chi‑square, ANOVA F‑test, mutual information), wrapper (recursive feature elimination, importance evaluation), and embedded methods (L1‑penalized logistic regression, linear SVM, tree‑based models). Example of recursive feature elimination:

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
X_selected = RFE(estimator=LogisticRegression(), n_features_to_select=10).fit_transform(X, y)

Feature Transformation covers standardization (Z‑score), normalization (L1/L2 norm), scaling (MinMax, MaxAbs), logarithmic and Box‑Cox transformations, and encoding techniques (LabelEncoder, OneHotEncoder, LabelBinarizer). Example of log transformation:

df_titanic['fare_log'] = np.log1p(df_titanic['fare'])

Dimensionality Reduction techniques such as PCA, SVD, LDA, and t‑SNE are demonstrated on the Iris dataset, showing how to project high‑dimensional data into two dimensions for visualization.

Finally, the article emphasizes best practices: perform thorough EDA, avoid feeding raw features directly into models, handle missing and outlier values based on statistical principles, apply appropriate scaling only when required, and prefer feature engineering over blind dimensionality reduction for most machine‑learning pipelines.