Abstract

—Conventional bias audits in facial recognition often attribute performance disparities to predefined demographic categories, such as gender and race. However, this category-based approach risks conflating correlation with causation and can obscure the actual feature-level causes of model errors. To address this gap, we analyzed misclassification patterns in a FaceNet model across four scenarios combining two datasets—CelebA (in-the-wild) and FERET (controlled)—with two preprocessing methods: imprecise cropping and precise MTCNN cropping. Using Grad-CAM visualizations, we examined how the dataset characteristics and intra-person variability shaped the model attention. Heatmap annotations were analyzed without presuming demographic categories as the error sources. The results revealed that uncontrolled feature variability—not demographic identity—is the primary driver of errors. While FERET (controlled) achieved over 96% accuracy, CelebA (in-the-wild) dropped to ∼60%. Preprocessing had a minimal overall impact, showing inconsistent effects across datasets, which led to minor accuracy gains or losses. Although error rates were initially higher for female subjects in CelebA, this disparity was strongly linked to their greater intra-person variation in attributes such as hairstyle. In fact, the gender gap disappeared in controlled settings (FERET) and decreased significantly (9%) in CelebA when transient attributes were excluded from the analysis. These findings challenge the prevailing assumption that demographic identity is a primary causal factor of bias, suggesting that this demographic focus may itself constitute a methodological bias. We demonstrated that recognition errors arise mainly from feature instability due to uncontrolled datasets and within-person variations. Thus, effective bias mitigation should shift the focus from demographic parity to improving model robustness in the face of natural human variability. ©2025 IEEE.