Case: German Credit

The German Credit data set (available at ftp.ics.uci.edu/pub/machine-learning-databases/statlog/) contains observations on 30 variables for 1000 past applicants for credit. Each applicant was rated as “good credit” (700 cases) or “bad credit” (300 cases).

New applicants for credit can also be evaluated on these 30 “predictor” variables. We want to develop a credit scoring rule that can be used to determine if a new applicant is a good credit risk or a bad credit risk, based on values for one or more of the predictor variables. All the variables are explained in Table 1.1. (Note: The original data set had a number of categorical variables, some of which have been transformed into a series of binary variables so that they can be appropriately handled by XLMiner. Several ordered categorical variables have been left as is; to be treated by XLMiner as numerical. The data has been organized in the spreadsheet German CreditI.xls)

Table 1.1    Variables for the German Credit data.

Table 1.2, below, shows the values of these variables for the first several records in the case.

Table 1.2   The data (first several rows)

The consequences of misclassification have been assessed as follows: the costs of a false positive (incorrectly saying an applicant is a good credit risk) outweigh the benefits of a true positive (correctly saying an applicant is a good credit risk) by a factor of five.    This can be summarized in the following table.

Table 1.3 Opportunity Cost Table (in deutch Marks)

The opportunity cost table was derived from the average net profit per loan as shown below:

Table 1.4 Average Net Profit

Let us use this table in assessing the performance of the various models because it is simpler to explain to decision-makers who are used to thinking of their decision in terms of net profits.

Assignment

Review the predictor variables and guess from their definition at what their role might be in a credit decision. Are there any surprises in the data?   — see additional columns to the right in the codelist sheet

Divide the data randomly into training (60%) and validation (40%) partitions, and develop classification models using the following data mining techniques in XLMiner:

Logistic regression

Classification trees

Neural networks

Choose one model from each technique and report the confusion matrix and the cost/gain matrix for the validation data. For the logistic regression model use a cutoff “predicted probability of success” (“success”=1) of 0.5. (for neural net and classification tree, see the “validation misclassification” part of the output; for logistic regression a probability cutoff must be specified and the cost/gain matrix constructed by hand. These tables are at the top left of LR_ClassifyValid2) Which technique gives the most net profit on the validation data? All ended up in negative territory (net costs). Logistic regression = -3200 DM, classification tree = -13,700 DM, and neural net –3800 DM

4. Let's see if we can improve our performance by changing the cutoff. Rather than accepting XLMiner's initial classification of everyone's credit status, let's use the “predicted probability of success” in logistic regression as a basis for selecting the best credit risks first, followed by poorer risk applicants.

a. Sort the validation data on “predicted probability of success.”

b. For each case, calculate the expected net profit of extending credit.

c. Add another column for cumulative expected net profit (this defines a “lift” curve in terms of money, rather than as a number or percentage).

d. How far into the validation data do you go to get maximum net profit? (Often this is specified as a percentile or rounded to deciles.) — 4^th or 5^th decile; see first three columns in LR_ClassifyValid2

e. If this logistic regression model is scored to future applicants, what “probability of success” cutoff should be used in extending credit? — approximately 0.8