Exploring effective feature selection methods in logistic regression for credit scoring models in XYZ bank

Abstract

This research explores optimized variable selection in credit scoring models, comparing wrapper, embedded, none, and hybrid approaches. Logistic regression is applied to model the dependence of explanatory variables on the target variable while mitigating underfitting, overfitting, and computational challenges. Feature selection methods help mitigate these issues by reducing dimensionality and improving model generalization. The Hybrid model achieved the best overall performance with accuracy = 0.9840 on the test set and accuracy = 0.9839 on the training set (test-train difference = -0.0001), as well as the best recall = 0.9993, F1-score = 0.9842, and the least number of false positives (FP = 47) and false negatives (FN = 1), which suggests good predictive ability and stability with the fewest misclassifications. The wrapper and embedded models have lower accuracies of 0.9793 and 0.9830, respectively, and mild overfitting. The baseline model with no feature selection performed the worst in terms of both accuracy and F1-score (accuracy = 0.5000, F1-score = 0.6667). In conclusion, the results indicate that the combination of wrapper and embedded approaches is the most accurate, stable, and generalisable model for credit scoring tasks.