In line with ECB instructions, validation test metrics are defined for following categories in each risk parameter:
- Qualitative tests: Summary statistics for IRB parameters and problematic data entries (outliers, technical defaults, etc.)
- Predictive ability: Assessment of the calibration accuracy of the model by comparing the estimated and realized values [Further potential test metrics: Vasicek test, Binomial test]
- Discriminatory power: Analysis of the rank-ordering ability of the model [Further potential test metrics: KS-statistic, Spearman’s rank correlation]
- Stability: Comparison of the risk characteristics of the portfolio (i.e. model application scope) at the beginning and end of a given observation period
- Concentration: Assessment of variation in risk parameter estimates
Finalyse validation toolkit can also be easily adjusted and expanded to include additional metrics to comply with institutions’ internal validation standards if required. The validation toolkit already contains some additional metrics that are commonly used in banking industry covering additional test categories such as homogeneity, heterogeneity or sensitivity. These test metrics can be used for internal validation reports, although they are not mentioned in ECB reporting instructions. In general, internal validation reports for all IRB risk parameters can also be automatized in a similar way by aligning the Finalyse validation toolkit with internal validation standards.