ECB’s Revised Guide to Internal Models: The Use of Machine Learning Techniques in Internal Models

The ECB defines ML techniques as:

• Models which rely on a large number of parameters, can represent highly non-linear functions and/or may require large amounts of data for their calibration (training). 
• Techniques that are well adapted to process unstructured data, such as text. 
• Techniques that are characterised as being highly complex and presenting limited explainability / interpretability.
   

• Governance frameworks and processes. Institutions should reflect the risks stemming from the use of ML-based internal models in their governance frameworks and processes. 
• Skills, capabilities and expertise.All the stakeholders (i.e. all three lines of defence, including compliance and the risk control unit as the second line and internal audit as the third line) should possess the necessary skills, capabilities and expertise.
• Model change. The initial switch from a largely traditional approach to a mostly ML-based approach should be classified as a material change. Where using dynamic ML-based internal models or components, institutions should justify the reason and effectively monitor the evolution at an appropriate frequency. 
• Model assessment. The use of ML techniques should be viewed as a driver in their complexity assessment and materiality assessment. ML-based internal models are more likely to be classified as material, with subsequently higher expectations towards management reporting and internal validation.

• Data assessment standards. Institutions should define standards for assessing the adequacy of the data types, amounts and sources used as inputs  for ML-based internal models. 
• Additional data analysis. Input data for ML techniques should undergo an additional exploratory analysis to ensure a clear understanding of data formats, the treatment of missing values, and potential sources of bias.

• Methodology justification. The methodology should be adequately justified and, where appropriate, based on statistical/optimality criteria.
• Potential bias consideration. Due to the ability of ML algorithms to fit the training data sample beyond their generalisation capabilities, the distribution of the prediction of an ML technique on the training data sample will usually differ from the distribution based on the overall population. Institutions should consider the potential bias on the estimates stemming from over- and under-fitting and choose an appropriate level of model complexity.
• Hyperparameters identification. Institutions should identify all relevant hyperparameters of their ML components. Unless otherwise justified, the determination of their values should be based on the model’s ability to generalise. 
• Documentation for replicability. The documentation of ML components should allow for their replication, including a determination of the parameters and hyperparameters. 

• Appropriate complexity. Institutions need to consider any trade-off between the complexity of the modelling techniques and the performance increase of the ML-based internal models.
• Justification of risk drivers. The inclusion of a large number of risk drivers needs proper justification. Institutions need to ensure the economic rationale of ML-based internal models and their risk drivers.
• Use of explainability techniques and tools. 
  • Functionality. Institutions are expected to rely on a set of explainability techniques and tools to support the plausibility and intuitiveness of their estimates. It should enable: 
    • a) an assessment and quantification of the contribution made by the individual risk drivers to the model prediction on a global level (i.e. across all predictions), and the relationship between the output of the model and each risk driver; 
    • b) the contribution of the individual key risk drivers to each given prediction.
  • Documentation. Institutions should define and document their use of the explainability techniques and tools, including the dimensions and criteria, their weaknesses and limits
  • Results. The explanation results should be robust, accurate and actionable, so as to permit informed human judgement and well-founded human oversight of the pre-overrides model outputs.

• Scope and purpose. Institutions should explicitly define the scope and purpose of the use of ML-based internal models in their risk management and decision-making processes and in their credit approval, internal capital allocation and corporate governance functions. 
• Documentation on exceptions and discrepancies. Where institutions use internal ratings or risk estimates resulting from ML-based internal models, if (part of) ML-based model is not used in specific areas, they should provide adequate reasons in the documentation for this choice, thus ensuring that any discrepancies are explained in a sound and understandable manner.
• Use in ICAAP. When an institution uses ML-based internal models in ICAAP, it should ensure that model outcomes in stress tests and scenario analyses are explainable, plausible and not overly optimistic.

• Indicator of model appropriateness. When observing excessive numbers/extents of overrides and/or an accumulation of inappropriate justifications, it should be considered as a strong indicator questioning the design and assumptions of the ML-based internal model, especially with regard to its complexity.
• Override monitoring. Institutions should implement the monitoring of overrides, including an assessment of the impact of the overrides on the rating model performance. This monitoring should distinguish between input and output overrides.
• Override documentation. Institutions must document and store sufficient information on all the overrides they conduct.
• Use of explainability techniques and tools. Institutions should consider the relevant explainability techniques and tools before conducting an override. They need to document the explainability techniques and tools considered for each rating override.

• Requirements. The internal validation should:
  • challenge the modelling decisions taken regarding ML-based internal models; 
  • assess whether the model complexity is justified; 
  • evaluate the stability and robustness of the outcomes, the hyperparameters selected and the randomness of the training process;
  • assess if the ML techniques fit the training dataset beyond its generalisation capacities (including overfitting); and 
  • assess the model performance using out-of-sample and out-of-time data. 
• Frequency. The validation frequency should be commensurate with the ML-based internal model’s complexity and model risk, to ensure that any material deficiency is detected within a reasonable time period. 
• Use of explainability techniques and tools. The internal checks are expected to rely on a set of explainability techniques and tools, to reliably identify:
  • circumstances (including instances) where an ML-based internal model is not working effectively;
  • the causes for a deterioration in the model’s performance and stability;
  • the reasons for the deviation in the risk estimates from their realised values. 
• Assessment of explainability techniques and tools. The explainability techniques and tools used in conjunction with the ML-based internal models should also be assessed. These analyses should be performed at least yearly, addressing in particular the adequacy and appropriateness of the chosen explainability techniques and tools.

• Higher model risk consideration. When performing the general risk assessment and drawing up the work plan, the institution should consider whether the use of ML-based internal model leads to higher model risk. 
• Audit engagements. Internal audit should set an appropriate intensity and frequency for the audit engagements of ML-based internal models. 

Institutions are now facing a strict and comprehensive regulation for ML-based models. To comply with the regulatory expectations, here are some key challenges and the corresponding recommended actions:

To summarize, the ECB’s revised guide to internal models provides a comprehensive guidance for internal models using ML techniques. Given the complexity of machine learning, the institutions are facing challenges in terms of model governance, documentation, explainability, and model use. To comply with the regulatory expectations, institutions need to take immediate actions to refine and enhance the model management framework when using ML techniques in internal models.

At Finalyse, our goal is to support banks and financial institutions in managing the challenges around ML models and regulatory expectations. Our team has significant expertise in both regulatory and non-regulatory models with deep knowledge of supervisory requirements.

What we offer to you:

• Experienced professionals – our consultants have years of practical and theoretical experience in applying ML models in different environments.
• Clear guidance on governance and transparency – we help you meet supervisory requirements with confidence.
• Training and workshops – we equip your teams with the skills to understand, interpret, and apply ML models responsibly in line with regulatory expectations.

Our expertise includes:

• Optimising model development through our in-house ML framework.
• Addressing interpretability challenges to maintain transparent and explainable models.
• Establish clear assessment criteria for model approach selection, providing the support through the full model cycle (e.g. model development, monitoring, validation, etc.)
• Sharing know-how through practical methods and real-life case studies

If you would like to talk about these points, we would be glad to discuss how we can support your institution.