Credit Risk Scorecards by Naeem Siddiqi (2005)

Introduction

As I enter a new industry, it will be vital for my success to be a sponge for the near future. Fortunately, my manager has been very forthcoming with the ins and outs of the financial lending industry and gave me Credit Risk Scorecards by Naeem Siddiqi. My goal with this book is threefold: (1) capture the most important information, (2) create a list of questions that will let me understand the industry and the job further, and (3) create a model emulating a credit risk scorecard.

The Seven Stages at a Glance

Stage 1: Preliminaries and Planning

• Create business plan
• Identify organizational objectives and scorecard role
• Determine internal versus external development and scorecard type
• Create project plan
• Identify project risks
• Identify project team and responsibilities

Stage 2: Data Review and Project Parameters

• Data availability and quality
• Data gathering for definition of project parameters
• Definition of project parameters
• Performance window and sample window
• Performance categories definition (target)
• Exclusions
• Segmentation
• Methodology
• Review of implementation plan

Stage 3: Development Database Creation

• Development sample specifications
• Sampling
• Development data collection and construction
• Adjustment for prior probabilities (factoring)

Stage 4: Scorecard Development

• Exploring data
• Identifying missing values and outliers
• Correlation
• Initial characteristic analysis
• Preliminary scorecard
• Reject inference
• Final scorecard production
• Scorecard scaling
• Choosing a scorecard
• Validation

Stage 5: Scorecard Management Reports

• Gains tables
• Characteristic reports

Stage 6: Scorecard Implementation

• Pre-implementation validation
• Strategy development: scoring strategy, setting cutoffs, policy rules, overrides

Stage 7: Post-implementation

• Scorecard and portfolio monitoring reports: scorecard management reports and portfolio performance reports

Stage 1 Notes: Preliminaries and Planning

Why adopt a "scorecard" format?

• This format is the easiest to interpret, and it appeals to a broad range of risk managers and analysts who do not have advanced knowledge of statistics or data mining.
• Reasons for declines, low scores, or high scores can be explained to customers, auditors, regulators, senior management, and other staff in simple business terms.
• The development process for these scorecards is not a black box and is widely understood, so it can easily meet any regulatory requirement on method transparency.
• The scorecard is very easy to diagnose and monitor using standard reports — analysts can perform these functions without in-depth knowledge of statistics or programming.

Examples of organizational objectives

• Reduction in bad debt/bankruptcy/claims/fraud
• Increase in approval rates or market share in areas such as secured loans, where low delinquency presents expansion opportunities
• Increased profitability
• Increased operational efficiency (e.g., to better manage workflow in an adjudication environment)
• Cost savings or faster turnaround through automation of adjudication using scorecards
• Better predictive power compared to an existing custom or bureau scorecard

Scorecard metrics should not be duplicated

The contents of the scorecard should preferably not be duplicated elsewhere. In a sole-arbiter setting, the scorecard should be based on as many independent data items as possible. In decision-support settings, the scorecard characteristics should complement the other considerations being used to evaluate credit — if policy rules are being used, the factors contained in those rules should preferably be kept out of the scorecard.

Identify project risks

• Non-availability of data or insufficient data
• Poor quality of data (dirty or unreliable)
• Delays or difficulties in accessing data
• Non-predictive or weak data
• Scorecard characteristics or derivations that cannot be handled by operational systems
• Changes in organizational direction/priorities
• Possible implementation delays
• Other legal or operational issues

A best practice worth remembering

For delinquency scorecard development, interview adjudication/collections staff during the project to get their input. A good question to ask: "What characteristics do you see in bad accounts, and have they changed over the last few years?"

Stage 2 Notes: Data Review and Project Parameters

Minimum necessary data

As a rule of thumb, for application scorecard development there should be approximately 2,000 "bad" accounts and 2,000 "good" accounts that can be randomly selected for each proposed scorecard, from a group of approved accounts opened within a defined time frame.

Is the internal data representative?

The project team needs to determine whether internal data intended for scorecard development has been tampered with or is unreliable for other reasons. Demographic data and other unverified application data items (such as income) are more susceptible to misrepresentation, but items such as credit bureau data, real estate data, and financial ratios are more robust.

Necessary data variables (previous 2–5 years)

• Account/identification number
• Date opened or applied
• Arrears/claims history over the life of the account
• Accept/reject indicator
• Product/channel and other segment identifiers
• Current account status (e.g., inactive, closed, lost, stolen, fraud)

Examples of data exclusion

Accounts with abnormal performance — for example, frauds — and those adjudicated using non-score-dependent criteria should not be part of any development sample. These can include staff, VIPs, out-of-country, pre-approved, lost/stolen cards, deceased, underage, and voluntary cancellations within the performance window. If there are geographic areas or markets where the company no longer operates, that data should also be excluded so the development data represents future expected status.

Establish performance

A simple way to establish performance and sample windows is to analyze payment or delinquency performance of the portfolio and plot the development of defined "bad" cases over time. A good source of this data is the monthly or quarterly cohort/vintage analysis report produced in most credit risk departments.

Questions I'm Bringing to My Team

Reading as an industry newcomer, the book generated a long list of questions I want to answer about how our own shop works. A sample:

1. Do we have an in-house adjudication/collections team? Who do we outsource to?
2. What types of delinquencies are there beyond failure to pay and fraud?
3. How is our fraud risk gauged, and does our team own it?
4. What is the application path — what steps does an application go through before a decision is rendered?
5. What are the pertinent laws and regulatory requirements?
6. Where do we get our data to train the models, and how do we know it's representative?
7. How do we protect against cohort or time-based effects?
8. How do we determine customer segmentation for each product — and do we use a separate scorecard per segment?
9. Do we impute missing variables, and do we track that they're imputed?
10. How do we detect a change in the population characteristics of a customer base, and what do we do in response?
11. What is our reject inference method?
12. Do we have to give borrowers reasons for declining their application?
13. Do we ever adjust the scorecard manually when it doesn't line up with experience?
14. Why does the book recommend error rate, sensitivity, and specificity over precision, recall, and F1 to gauge misclassification?
15. When choosing which scorecard to proceed with, how do we decide whether to minimize false positives or false negatives? Minimizing false negatives (predicting someone won't default when they do) increases risk and loss, while minimizing false positives (predicting default when they wouldn't) decreases risk but leaves money on the table. Since we know the range that we lend out and the total liability the company has, could we not calculate a numerical amount of risk we're willing to take on?