A feature selection algorithm, as a preprocessing method of model creation, is used in the process of eliminating features from a data set that are irrelevant to the task being performed (Kumar 2014; Chandrashekar and Sahin 2014; Guyon et al 2003). The major benefits of such an algorithm are as follows.

1. (1)

   It facilitates one’s understanding of data since it finds useful features to represent the data and removes the irrelevant features.

2. (2)

   It reduces computational time by reducing the dimensionality of data.

3. (3)

   It improves model quality and the interpretability of the outcome.

In general, there are three major types of feature selection approaches are applied to credit scoring (Miao and Niu 2016; Jiménez et al 2016). The first is the filter approach, which selects variables by ranking them with information generated from data and independent of the classification model. Filter methods that are often used in credit scoring include the information gain ratio (Wang et al 2017), mutual information (Zhang et al 2018), rough set (Ping and Lu 2011), principal component analysis (PCA) (Šušteršič et al 2009) and LDA (Chen and Li 2010). The second is the wrapper approach, which assesses a subset of features based on their performance with a given classifier model such as SVMs or LR models. Typical methods include the genetic algorithm (GA) (Koutanaei et al 2015), the wrapper method based on the genetic algorithm (Ala’raj and Abbod 2016) and recursive feature elimination (Bellotti and Crook 2009). The third is the embedded approach, in which feature selection is embedded in the classifier. The most typical models are DT-based approaches (Wang et al 2017), such as the classification and regression tree (CART) and C4.5 algorithms.

The studies cited above have demonstrated that using feature selection methods in credit scoring can improve prediction performance. However, as illustrated in Zhang et al (2018), different kinds of feature selection models have their own advantages and disadvantages, and there is no overall best feature selection method for the credit scoring problem. Currently, researchers are attempting to combine different kinds of feature selection methods to achieve better performance, and the studies by Chen and Li (2010) and Zhang et al (2018) have shown that the combination approach is superior to a single feature selection method.

Ensemble learning is a machine learning paradigm in which multiple learners are trained to solve the same problem. A classifier ensemble contains a number of individually trained base classifiers (eg, SVMs, ANNs) whose decisions are combined for use, and when classifying new samples, the most popular combination schemes are weighted or unweighted. There are many ensembles that have been applied to solving the credit scoring problem, and the bagging (Sun et al 2014; Hsieh et al 2010), boosting (Sun et al 2016; Paleologo et al 2010; Bian and Wang 2007), random subspace (Marqués et al 2012; Nanni and Lumini 2009), rotation forest, random forest (Ala’raj and Abbod 2016; Twala 2010) and GBDT (Ma et al 2018) methods are currently the most extensively used ensembles.

The works cited above have verified the superior stability and learning results of these models compared with single base classifiers. However, when dealing with the credit scoring problem, it remains an open question of which ensemble performs best. Recently, some researchers have begun to compare the effectiveness of ensembles. Lessmann et al (2015) compared individual base classifiers, homogeneous ensembles (eg, bagging, random forest) and heterogeneous ensembles (eg, dynamic ensembles) based on several credit data sets, including Australian, German, Benelux and UK credits, and they concluded that the random forest method achieves promising performance. García et al (2019) compared seven ensembles with different data sets, such as Australian, Finnish and German credits, and their results revealed that the performance of ensembles depends on the sample types in the data set; therefore, comparisons between various ensembles should be discussed with regard to a specific data set. The studies by Feng et al (2018) and Wang et al (2011) constitute similar research. However, most current studies focus on data sets in developed countries such as the United States, Australia and Germany, while minimal attention has been paid to the Chinese P2P market.²² 2 The reason may be that the Chinese P2P data are not open source. Moreover, to the best of our knowledge, comparisons between the six popular ensembles listed above are still lacking in the Chinese context.

In this study, we propose a hybrid system to find the most superior ensemble for Chinese P2P lending services. The system first selects the most relevant features by combining three kinds of typical feature selection methods. Subsequently, the selected features are fed into six popular ensembles with five base classifiers, and the results are then compared from standard and statistical perspectives.

The end goal of this step is to find a subset of features that offers the highest discriminative power for the next step. As demonstrated in Section 2.1, there is a trend of developing a hybrid feature selection method to realize an advantageous combination. Here, we combine the three feature selection algorithms listed in Section 3.1. The combination scheme is as follows: a feature $F_i$ receives three scores $\{S(F_i{}^{\mathrm{fs1}}),S(F_i{}^{\mathrm{fs2}}),S(F_i{}^{\mathrm{fs3}})\}$ based on the three algorithms $\mathrm{fs1}$, $\mathrm{fs2}$ and $\mathrm{fs3}$; then, the final score of each feature $SF\{F_i\}$ is computed by an average voting mechanism as follows:

Subsequently, the 10 features with the highest final scores are selected as the final results; the features are listed in Table 5. The reasons why we choose 10 features are as follows. First, when we calculate the 10-fold cross-validation $F$-measure⁴⁴ 4 The reason we choose the $F$-measure as the evaluation metric rather than accuracy is that the data set is highly imbalanced, as illustrated in Section 4.3. of the top $k$ features ranked by the information gain ratio algorithm with the SVM classifier, some features will likely be noise, and the optimal feature number is 10 since it gives the best performance (see Figure 3). Second, the number of features selected by the GA approach is 10. Third, when comparing the top 10 selected features of the three methods (see Table 6), there are no significant differences. Moreover, when comparing the performance of the three single feature selection approaches with our combination approach using the SVM classifier, the results shown in Table 6 clearly verify the superiority of our combined approach. Thus, the top 10 features are used in the following sections.

A data set is said to be imbalanced if the number of samples from one class is higher than that from another. The class with more instances is called the major class, while the class with fewer instances is called the minor class (Longadge and Dongre 2013; Chawla et al 2004). As illustrated in Phase I, the number of “good” loans (major class) in our data set is 33 492, which is much higher than the number of “bad” loans (minor class), which is 474. Thus, it is easy to obtain that the imbalance rate is 70.66, which means that the data set is highly skewed.

As illustrated by many researchers (see, for example, Shen et al 2019), an uneven distribution of samples will lead to skewed results due to data bias toward the major class, and this issue challenges existing models. Moreover, a highly skewed data set can lead to misleading results for some typical evaluation metrics. When facing a skewed data set, it is easy to understand that the accuracy metric will obtain biased results. Even receiver operator characteristic (ROC) curves, which are often employed to evaluate imbalanced data sets, can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution (Davis and Goadrich 2006).

Therefore, when facing the highly skewed data set in this study, preprocessing the imbalanced data set before modeling is highly important. As a simple and effective oversampling method, the SMOTE algorithm proposed by Chawla et al (2002) is employed in this study to handle the imbalance issue. Based on this algorithm, there exists a virtual sample between two real samples that are near one another. Therefore, in this study, the SMOTE algorithm artificially invents a new “bad” loan between the two real “bad” loans that are near each other based on the following two steps.

Step 1. Given the unbalanced training data set $D$, $C_{\mathrm{bad}}\subset T$ represents the minority class (“bad” loans), and for each sample of $x_i\in C_{\mathrm{bad}}$, the $k$ nearest neighbors ($\{x_{i1},\mathrm{\dots },x_{ib},\mathrm{\dots },x_{ik}\}$) are identified. In this paper, $k$ is set to a default value 5 based on previous studies (Maldonado et al 2019).

Step 2. A new “bad” loan $x_{ib}^{\mathrm{new}}$ around the original sample $x_i\in C_{\mathrm{bad}}$ can be artificially invented with the randomly selected neighbor $x_{ib}$ based on the following formula:

where $\delta$ denotes the random value in (0, 1).

In this manner, the procedure for synthesizing the instance of the minority class is repeated until the training data set is balanced. After the SMOTE treatment, there are 46 888 loans in the final training data set, where there are 23 444 instances of both “good” and “bad” loans.

As described in Section 3.2 and Figure 1, six ensembles are tested in this paper – the bagging, boosting, random subspace, rotation forest, random forest and GBDT methods – with the following five popular base classifiers: LR, CART, C4.5, multilayer perceptron (MLP) neural networks with a back-propagation algorithm and SVMs with a linear kernel. The bagging, boosting, random subspace and rotation forest methods are implemented with the Waikato environment for knowledge analysis (WEKA) open-source data mining toolkit (Witten and Frank 2002; Witten et al 2011), while the random forest and GBDT methods are performed with the Sklearn package in Python 3.6.

In practice, to improve the forecasting performance of these models, a few parameters need to be optimized before classifier construction. In this study, the grid search method is used to determine the optimal parameters that generate minimum forecasting errors. The optimized results are listed in Table 7.

For base classifiers, the detailed parameter tuning processes are as follows.

CART and C4.5 algorithms.

The test range of a minimal number of samples at the terminal leaf is from 2 to 10. In addition, the number of features is set to 10.

MLP method.

A grid search is carried out to find the optimal number of hidden layers and the learning rate. The search ranges are [1,10] and [0.001,1], respectively.

SVMs.

Various cost hyperparameters $C$ and $\gamma$ are evaluated. Moreover, different kernel types are tested, and the candidates are linear, polynomial, radial basis function and sigmoid.

LR and MLP.

The cutoff values are set at a default value of 0.5.

For ensembles, the detailed parameter tuning processes are as follows.

Random forest method.

Different “number of trees” and “number of attributes”, used to grow each tree, are tested. The “number of trees” is adjusted from 10 to 100 with nine steps and the “number of attributes” is tested from 1 to 10.

GBDT method.

Different “number of iterations”, “learning rate” and “number of attributes” are tested. The “number of iterations” is tuned from 10 to 100 with nine steps, the “learning rate” is adjusted in the range [0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.5, 0.8, 1] and the “number of attributes” is tested from 1 to 10.

Bagging, boosting, random subspace and rotation forest methods.

The default settings in the Weka package are adopted (Abellán and Castellano 2016; Marqués et al 2012).

To evaluate the performance of various methods, two kinds of evaluation procedures are employed. The first involves standard evaluation metrics based on the confusion matrix described in Table 8; here, the “bad” samples are chosen as the positive class.⁵⁵ 5 As illustrated in Tan et al (2005), the minority class can be chosen as the positive class due to its high importance. The corresponding metrics are as follows.

• •

  Accuracy = $(\mathrm{TP}+\mathrm{TN})/(\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN})$, which represents the correctly discriminative samples, including “bad” and “good” loans.

• •

  Type I error = $\mathrm{FN}/(\mathrm{TP}+\mathrm{FN})$, which is the proportion of actual “bad” loans predicted to be “good”. Given its high probability of causing high costs, we must pay attention to this metric.

• •

  $F$-measure = $2\mathrm{T}\mathrm{P}/(2\mathrm{T}\mathrm{P}+\mathrm{FP}+\mathrm{FN})$, which represents a type of harmonic mean for precision and recall, where precision is the fraction of true positive samples among the samples that the model classified as positive ($\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$), and where recall is the fraction of samples classified as positive among the total number of positive samples ($\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$). As illustrated in Soleymani et al (2020), widely used evaluation metrics, such as accuracy and the area under the ROC curve (AUC), tend to favor the correct classification of the most populated class, which will, together with imbalanced test data, cause biased high scores. To address this issue, in this paper we add the $F$-measure evaluation metric, which pays more attention to the minority class. For details, see Thomas et al (2017).

To compare the performance of various ensembles from a statistical perspective, the Friedman test (Friedman 1940; Demšar 2006), which is a rank-based nonparametric test, is employed to test the statistical significance of differences in performance. The Friedman test is distributed based on the chi-square distribution with $m-1$ degrees of freedom, where $m$ is the number of methods (in this study, ensemble methods). The corresponding statistic is computed as follows:

where $r_j^i$ denotes the individual rank of each method $i$ on each data set $j$, and $h$ is the number of data sets (according to Section 5, there are three different data sets: one original data set and two robust testing data sets, which means that in this study $h=3$). If ${\chi }_F^2$ is larger than a critical value, the null hypothesis that all the methods are equivalent is rejected. Then, a post hoc test, specifically the Bonferroni–Dunn test (Dunn 1961), is employed to compare all the ensembles (eg, $i$) with a control ensemble $k$. The statistic for comparing methods $i$ and $k$ is computed as follows.

where $q_{p,\mathrm{\infty },m}$ is computed based on the $t$-test statistic with a confidence level $p/(m-1)$. If the average ranks of methods $i$ and $k$ differ by at least the critical difference (CD), there is a significant difference in the performance of methods $i$ and $k$.

The results of the discriminative performance of the six ensembles⁶⁶ 6 Here we delete the bagging results with the LR and SVM base classifiers because the two strong base classifiers are not suitable for bagging, which focuses on reducing variance. on the testing set with full features and selected features as measured by accuracy, Type I error and $F$-measure are listed in Table 9 and Figure 4. Table 9 lists in detail the results of different ensembles with corresponding base classifiers, and Figure 4 shows a summary of the average performance⁷⁷ 7 Here the average refers to the mathematical average performance of different base classifiers by the bagging, boosting, random subspace and rotation forest methods. of the three measures. Importantly, we also compare the performance of ensembles and the grade feature provided by the lending platform. According to the platform, this feature is generated in the loan application process by evaluating the application documents, and the grade can be divided into seven levels (from low to high): HR, E, D, C, B, A and AA. Here, we select the lowest grade, HR, as the reference for “bad” loans and select the others as “good” loans; the results are listed in Figure 4. Moreover, to discover the effect of feature selection, we also compare the performance of all features (see Table 2) and the top 10 selected features (see Table 5).

As shown in Table 9 and Figure 4, we obtain the following findings. First, it is easy to see that the performances of the six ensembles are at different grades. The first grade includes the GBDT, random forest and rotation forest methods, which are powerful ensembles and hold the top three spots in terms of the three different evaluations. For example, as illustrated in Figure 4, with the top 10 features, the GBDT method achieves the highest $F$-measure (approximately 0.94), which is approximately 13% better than that achieved by the random subspace method (approximately 0.8). The second grade includes the bagging and boosting ensembles, which have similar performance. The third grade consists of the random subspace ensemble, which achieves the worst performance in terms of the three evaluation metrics. The detailed information listed in Table 9 shows that compared with other ensembles, the random subspace ensemble has the largest Type I error, which may be the key reason for the poor performance of random subspace.

Second, the grade feature has better performance than the random subspace ensemble in terms of different evaluation metrics and different features. Compared with the bagging and boosting ensembles, interestingly the grade feature achieves a better Type I error score but worse accuracy and $F$-measure scores. The reason behind this result may be that to avoid risk, the platform tends to give low grades.

Third, each ensemble with the 10 selected features yields better performance than the ensemble with all 22 features, whether measured by detailed evaluations with base classifiers or average comparisons. This finding clearly indicates the effectiveness of our combined feature selection approach and the usefulness of the top 10 selected features.

Finally, regarding the Type I error metric, interestingly, the ensembles with the selected features can greatly reduce this error. For instance, Table 9 shows that even random subspace, the worst performing ensemble, combined with the C4.5 base classifier, improves performance by 9% on the $F$-measure with the selected features compared with performance that includes all features. In conclusion, the selected features show that they are capable of discriminating successfully between “bad” loans and “good” loans.

To evaluate the effectiveness and robustness of the performance results listed above, we further compare the six ensembles with out-of-time instances. Moreover, we list the performance results with a different data partition ratio: 50% for training, 20% for validation and 30% for testing. The results are listed in the online supplementary materials.

Here, the out-of-time loan data are collected from January 2017 to June 2019, and the loans that appear in the former data set (2013–16) are deleted from this data set. The final number of instances (considering only “fully paid” and “charge off”) is 39 201, including 607 “bad” loans. To maintain consistency with the former comparison, the data are divided into three parts: 60% for training, 20% for validation and 20% for testing.

Table 10 and Figure 5 show the detailed and average performance of the six ensembles and the grade feature on the testing set. It is easy to see that the results are consistent with those of the previous experiment. For instance, Table 10 clearly shows that the GBDT method performs best, followed by the random forest and rotation forest methods, while the random subspace method still has the worst performance. Moreover, the grade feature has better performance than the random subspace ensemble. Interestingly, although it still performs worse than the other ensembles, it has better scores than those it obtained with the former data set listed in Section 5.1, which may imply that the P2P platform has achieved progress in identifying “bad” loans. In addition, Figure 5 clearly shows that the ensembles with the selected features outperform those with all features. Accordingly, we conclude that the comparison results are robust to different time periods.

The nonparametric Friedman test and the post hoc (Bonferroni–Dunn) test were employed to test the statistical significance of the differences in performance. In this section, we also consider the values of Friedman’s ranks for the accuracy and AUC measures. Performance with two different training, validation and testing ratios (ie, 60:20:20, 50:20:30) and out-of-time data sets are used to conduct the tests. The rank values represent an important reference for performance when several ensembles are compared and can be considered in order of their performance.

First, the Friedman test statistic is computed to verify whether the ensembles exhibit different prediction performances. The results are 35.59 and 35.47 on the two measures, both rejecting the null hypothesis that the ensembles exhibit no different performances at the significance level of 99.5%. Then, the Bonferroni–Dunn test is applied to conduct comparisons. The CDs at significance levels of 0.05 and 0.1 are computed as 10.532 and 9.781, and the average ranks of various ensembles and the grade feature are shown in Figure 6. Thus, the following findings can be obtained. First, the GBDT, random forest and rotation forest ensembles are generally the best. More specifically, the GBDT method with the top 10 features has the lowest rank in terms of two different metrics. Second, the grade feature holds the middle position among all of the ensembles. As shown in Figure 6, the average rank of the grade is 8, which means that it cannot beat the GBDT, random forest and rotation forest ensembles. Third, regarding the same ensemble with different numbers of features, it is easy to conclude that compared with the method with all features, the method with the top 10 selected features has the lower rank. Fourth, as shown in Figure 6, when the GBDT method with the top 10 features is used as the benchmark, the random subspace and bagging methods have average ranks that are higher than the CD, suggesting that in terms of accuracy, they are significantly worse than the GBDT approach at a significance level of 0.1. Additionally, this pattern is reproduced with the accuracy listed in the online supplementary materials, which further indicates that our results are robust.

In summary, the statistical tests also support the finding that the GBDT, random forest and rotation forest methods are generally the best for the Chinese P2P lending service since they outperform the other ensembles and the grade feature on various measures, different training-to-testing ratios and out-of-time samples. Moreover, the experimental results prove that the top 10 selected features can improve credit scoring performance, which also proves the effectiveness of the combined feature selection process in our proposed system.

Since the evaluation test results and statistical tests demonstrate the effectiveness of the selected features, we conduct a detailed analysis of the factors. Table 11 lists the importance of the features selected by the GBDT ensemble model. By comparing the contributions of the features, the following insights can be obtained.

1. (1)

   As illustrated in Table 12, the contributions of the five categories of features in the original data set from high to low are the characteristics of loans, the financial information of borrowers, the credit history of borrowers, the characteristics of borrowers and platform authentication information. These results indicate that the characteristics of loans have the greatest impact. Interestingly, one of the characteristics of borrowers, $F_{\text{1}}$ (age), has a competitive ranking (rank 5), while another feature, $F_{\text{3}}$ (marital status), obtains the lowest rank, which indicates that age contributes more than the marital status. The reason may be that in China, age represents a borrower’s economic strength, and a younger age may increase the default risk, as illustrated in Section 4.1.2.

2. (2)

   As depicted in Table 12, the ranks of the five categories of features in the 2017–19 data set (from high to low) are the characteristics of loans, platform authentication information, the financial information of borrowers, the credit history of borrowers and the characteristics of borrowers. It is easy to see that the rank of platform authentication information increases with the rank in original data set, possibly because it was easier to obtain loans at an early time (2013–16). Thus, some “good” borrowers do not provide effective authentication information. However, over time the platform authentication information, as a crucial indicator of repayment ability, becomes increasingly important. Thus, borrowers who do not provide effective authentication information are more likely to be “bad” borrowers.

3. (3)

   Comparing the findings listed above with the similar impact factor analysis in Ma et al (2018), which focuses on US P2P lending services, we can make the following interesting remarks. First, the high proportions of features holding high ranks in the characteristics of loans subset (similar to the borrowing details in Ma et al (2018)) are similar in the two platforms, which means that loan details are the most relevant information for the final loan result. Second, the two platforms differ in that there fewer financial information features (similarly to the financial situation in Ma et al (2018)) but more authentication information features in the Chinese P2P platform than the US one. The reason behind this may be that authentication information features are the benchmark for credit authentication; also, the credit authentication system in China is not as mature as that in the United States. Overall, the platform needs more corresponding features to capture the borrower’s credit situation.