Tag: model risk

In today’s complex business landscape, data-based decision-making has become the norm, with advanced technologies and analytics tools facilitating faster and more accurate modeling and predictions. However, with the increased reliance on models, the risk of errors has also increased, making it crucial for organizations to have a comprehensive model risk management framework. In this blog post, we will dive deeper into model risk management, its importance for organizations, and the key elements of a model risk management framework. What is model risk? First, let's define what we mean by model risk. Many institutions use models to forecast and predict the future performance of investments, portfolios or consumers' creditworthiness. Model risk can happen when the results produced by these models are inaccurate or not fit for the intended purpose. This risk arises due to several factors, like data limitations, model assumptions and inherent complexities in the underlying modeled processes. For example, in the credit industry, an inaccurately calibrated credit risk model may incorrectly assess a borrower's default risk, resulting in erroneous credit decisions and impacting overall portfolio performance. What is a risk management model and why is it important? A risk management model, or model risk management, refers to a systematic approach to manage the potential risks associated with the use of models and, more specifically, quantitative models built on data. Since models are based on a wide range of assumptions and predictions, it's essential to recognize the possibility of errors and acknowledge its impact on business decisions. The goal of model risk management is to provide a well-defined and structured approach to identifying, assessing, and mitigating risks associated with model use. The importance of model risk management for institutions that leverage quantitative risk models in their decisioning strategies cannot be overstated. Without proper risk management models, businesses are vulnerable to significant consequences, such as financial losses, regulatory enforcement actions and reputational damage. Model risk management: essential elements The foundation of model risk management includes standards and processes for model development, validation, implementation and ongoing monitoring. This includes: Policies and procedures that provide a clear framework for model use and the associated risks. Model inventory and management that captures all models used in an organization. Model development and implementation that documents the policies for developing and implementing models, defining critical steps and role descriptions. Validation and ongoing monitoring to ensure the models meet their stated objectives and to detect drift. In addition to these essential elements, a model risk management framework must integrate an ongoing system of transparency and communication to ensure that each stakeholder in model risk governance is aware of the policies, processes and decisions that support model use. Active engagement with modelers, validators, business stakeholders, and audit functions, among other stakeholders, is essential and should be included in the process. How we can help Experian® provides solutions and risk mitigation tools to help organizations of all sizes establish a solid model risk management framework to meet regulatory and model risk governance requirements, improve overall model performance and identify and mitigate potential risk. We provide services for back testing, benchmarking, sensitivity analysis and stress testing. In addition, our experts can review your organization’s current model risk management practices, conduct a gap analysis and assist with audit preparations. Learn more *This article includes content created by an AI language model and is intended to provide general information.

Model governance is growing increasingly important as more companies implement machine learning model deployment and AI analytics solutions into their decision-making processes. Models are used by institutions to influence business decisions and identify risks based on data analysis and forecasting. While models do increase business efficiency, they also bring their own set of unique risks. Robust model governance can help mitigate these concerns, while still maintaining efficiency and a competitive edge. What is model governance? Model governance refers to the framework your organization has in place for overseeing how you manage your development, model deployment, validation and usage.1 This can involve policies like who has access to your models, how they are tested, how new versions are rolled out or how they are monitored for accuracy and bias.2 Because models analyze data and hypotheses to make predictions, there's inherent uncertainty in their forecasts.3 This uncertainty can sometimes make them vulnerable to errors, which makes robust governance so important. Machine learning model governance in banks, for example, might include internal controls, audits, a thorough inventory of models, proper documentation, oversight and ensuring transparent policies and procedures. One significant part of model governance is ensuring your business complies with federal regulations. The Federal Reserve Board and the Office of the Comptroller of the Currency (OCC) have published guidance protocols for how models are developed, implemented and used. Financial institutions that utilize models must ensure their internal policies are consistent with these regulations. The OCC requirements for financial institutions include: Model validations at least once a year Critical review by an independent party Proper model documentation Risk assessment of models' conceptual soundness, intended performance and comparisons to actual outcomes Vigorous validation procedures that mitigate risk Why is model governance important — especially now? More and more organizations are implementing AI, machine learning and analytics into their models. This means that in order to keep up with the competition's efficiency and accuracy, your business may need complex models as well. But as these models become more sophisticated, so does the need for robust governance.3 Undetected model errors can lead to financial loss, reputation damage and a host of other serious issues. These errors can be introduced at any point from design to implementation or even after deployment via inappropriate usage of the model, drift or other issues. With model governance, your organization can understand the intricacies of all the variables that can affect your models' results, controlling production closely with even greater efficiency and accuracy. Some common issues that model governance monitors for include:2 Testing for drift to ensure that accuracy is maintained over time. Ensuring models maintain accuracy if deployed in new locations or new demographics. Providing systems to continuously audit models for speed and accuracy. Identifying biases that may unintentionally creep into the model as it analyzes and learns from data. Ensuring transparency that meets federal regulations, rather than operating within a black box. Good model governance includes documentation that explains data sources and how decisions are reached. Model governance use cases Below are just three examples of use cases for model governance that can aid in advanced analytics solutions. Credit scoring A credit risk score can be used to help banks determine the risks of loans (and whether certain loans are approved at all). Governance can catch biases early, such as unintentionally only accepting lower credit scores from certain demographics. Audits can also catch biases for the bank that might result in a qualified applicant not getting a loan they should. Interest rate risk Governance can catch if a model is making interest rate errors, such as determining that a high-risk account is actually low-risk or vice versa. Sometimes changing market conditions, like a pandemic or recession, can unintentionally introduce errors into interest rate data analysis that governance will catch. Security challenges One department in a company might be utilizing a model specifically for their demographic to increase revenue, but if another department used the same model, they might be violating regulatory compliance.4 Governance can monitor model security and usage, ensuring compliance is maintained. Why Experian? Experian® provides risk mitigation tools and objective and comprehensive model risk management expertise that can help your company implement custom models, achieve robust governance and comply with any relevant federal regulations. In addition, Experian can provide customized modeling services that provide unique analytical insights to ensure your models are tailored to your specific needs. Experian's model risk governance services utilize business consultants with tenured experience who can provide expert independent, third-party reviews of your model risk management practices. Key services include: Back-testing and benchmarking: Experian validates performance and accuracy, including utilizing statistical metrics that compare your model's performance to previous years and industry benchmarks. Sensitivity analysis: While all models have some degree of uncertainty, Experian helps ensure your models still fall within the expected ranges of stability. Stress testing: Experian's experts will perform a series of characteristic-level stress tests to determine sensitivity to small changes and extreme changes. Gap analysis and action plan: Experts will provide a comprehensive gap analysis report with best-practice recommendations, including identifying discrepancies with regulatory requirements. Traditionally, model governance can be time-consuming and challenging, with numerous internal hurdles to overcome. Utilizing Experian's business intelligence and analytics solutions, alongside its model risk management expertise, allows clients to seamlessly pass requirements and experience accelerated implementation and deployment. Experian can optimize your model governance Experian is committed to helping you optimize your model governance and risk management. Learn more here. References 1Model Governance," Open Risk Manual, accessed September 29, 2023. https://www.openriskmanual.org/wiki/Model_Governance2Lorica, Ben, Doddi, Harish, and Talby, David. "What Are Model Governance and Model Operations?" O'Reilly, June 19, 2019. https://www.oreilly.com/radar/what-are-model-governance-and-model-operations/3"Comptroller's Handbook: Model Risk Management," Office of the Comptroller of the Currency. August 2021. https://www.occ.treas.gov/publications-and-resources/publications/comptrollers-handbook/files/model-risk-management/pub-ch-model-risk.pdf4Doddi, Harish. "What is AI Model Governance?" Forbes. August 2, 2021. https://www.forbes.com/sites/forbestechcouncil/2021/08/02/what-is-ai-model-governance/?sh=5f85335f15cd

Financial institutions have long been on the cutting edge of technology trends, and it continues to be true as we look at artificial intelligence and machine learning. Large analytics teams are using models to solve for lending decisions, account management, investments, and more. However, unlike other industries taking advantage of modeling, financial institutions have the added complexity of regulation and transparency requirements to ensure fairness and explainability. That means institutions need highly sophisticated model operations and a highly skilled workforce to ensure that decisions are accurate and accountability is maintained. According to new research from Experian, we see that while financial institutions plan to use or are using models for a wide range of use cases, there is a range of ModelOps maturity across the industry. Just under half of financial institutions are in the early stages of model building, where projects are more ad-hoc in nature and experimental. Only a quarter of institutions seem to be more mature, where processes are well defined and models can be developed in a reliable timeframe. With more than two-thirds of lenders saying that ModelOps will play a key role in shaping the industry over the next five years, the race to maturity is critical. One of the biggest challenges we see in the space is that it takes too long for models to make it into production. On average, financial institutions estimate that the end-to-end process for creating a new model for credit decisioning takes an average of 15 months. Organizations need to accelerate model velocity, meaning the time that it takes to get a model into production and generating value, to take advantage of this powerful technology. Having the right technology, the right talent, and the right data at the right time continue to drag down operational speed and tracking of models after they are in production. For more information on Experian’s recent study, download the new report ‘Accelerating Model Velocity in Financial Institutions’. We are also hosting an upcoming webinar with tips on how to tackle some of the biggest model development and deployment challenges. You can register for the webinar here.

Machine learning (ML) is a powerful tool that can consume vast amounts of data to uncover patterns, learn from past behaviors, and predict future outcomes. By leveraging ML-powered credit risk models, lenders can better determine the likelihood that a consumer will default on a loan or credit obligation, allowing them to score applicants more accurately. When applied to credit decisioning, lenders can achieve a 25 percent reduction in exposure to risky customers and a 35 percent decrease in non-performing loans.1 While ML-driven models enable lenders to target the right audience and control credit losses, many organizations face challenges in developing and deploying these models. Some still rely on traditional lending models with limitations preventing them from making fast and accurate decisions, including slow reaction times, fewer data sources, and less predictive performance. With a trusted and experienced partner, financial institutions can create and deploy highly predictive ML models that optimize their credit decisioning. Case study: Increase customer acquisition with improved predictive performance Looking to meet growth goals without increasing risk, a consumer goods retailer sought out a modern and flexible solution that could help expand its finance product options. This meant replacing existing ML models with a custom model that offers greater transparency and predictive power. The retailer partnered with Experian to develop a transparent and explainable ML model. Based on the model’s improved predictive performance, transparency, and ability to derive adverse action reasons for declines, the retailer increased sales and application approval rates while reducing credit risk. Read the case study Learn about our custom modeling capabilities 1 Experian (2020). The Art of Decisioning in Uncertain Times

Artificial intelligence is here to stay, and businesses who are adopting the newest AI technology are ahead of the game. From targeting the right prospects to designing effective collections efforts, AI-driven strategies across the entire customer lifecycle are no longer a nice to have - they are a must.  Many organizations are late to the game of AI and/or are spending too much time and money designing and redesigning models and deploying them over weeks and months. By the time these models are deployed, markets may have already shifted again, forcing strategy teams to go back to the drawing board. And if these models and strategies are not being continuously monitored, they can become less effective over time and lead to missed opportunities and lost revenue. By implementing artificial intelligence in predictive modeling and strategy optimization, financial institutions and lenders can design and deploy their decisioning strategies faster than ever before and make incremental changes on the fly to adapt to evolving market trends.  While most organizations say they want to incorporate artificial intelligence and machine learning into their business strategy, many do not know where to start. Targeting, portfolio management, and collections are some of the top use cases for AI/ML strategy initiatives.  Targeting  One way businesses are using AI-driven modeling is for targeting the audiences that will most likely meet their credit criteria and respond to their offers. Financial institutions need to have the right data to inform a decisioning strategy that recognizes credit criteria, can respond immediately when prospects meet that criteria and can be adjusted quickly when those factors change. AI-driven response models and optimized decision strategies perform these functions seamlessly, giving businesses the advantage of targeting the right prospects at the right time.  Credit portfolio management  Risk models optimized with artificial intelligence and machine learning, built on comprehensive data sets, are being used by credit lenders to acquire new revenue and set appropriate balance limits. Strategies built around AI-driven risk models enable businesses to send new offers and cross-sell offers to current customers, while appropriately setting initial credit limits and managing limits over time for increased wallet share and reduced risk.   Collections  AI- and ML-driven analytics models are also optimizing collections strategies to improve recovery rates. Employing AI-powered balance and response models, credit lenders can make smarter collections decisions based on the most predictive and accurate information available.   For lending businesses who are already tight on resources, or those whose IT teams cannot meet the demand of quickly adapting to ever-changing market conditions and decisioning criteria, a managed service for AI-powered models and strategy design might be the best option. Managed service teams work closely with businesses to determine specific use cases, develop models to meet those use cases, deploy models quickly, and monitor models to ensure they keep producing and predicting optimally.  Experian offers Ascend Intelligence Services, the only managed service solution to provide data, analytics, strategy and performance monitoring. Experian’s data scientists provide expert guidance as they collaborate with businesses in developing and deploying models and strategies around targeting, acquisitions, limit-setting, and collections. Once those strategies are deployed, Experian continually monitors model health to ensure scores are still predictive and presents challenger models so credit lenders can always have the most accurate decisioning models for their business. Ascend Intelligence Services provides an online dashboard for easy visibility, documentation for regulatory compliance, and cloud capabilities to deliver scores and decisions in real-time.  Experian’s Ascend Intelligence Services makes getting into the AI game easy. Start realizing the power of data and AI-driven analytics models by using our ROI calculator below: initIframe('611ea3adb1ab9f5149cf694e'); For more information about Ascend Intelligence Services, visit our webpage or join our upcoming webinar on October 21, 2021.  Learn more Register for webinar

This is the third in a series of blog posts highlighting optimization, artificial intelligence, predictive analytics, and decisioning for lending operations in times of extreme uncertainty. The first post dealt with optimization under uncertainty and the second with predicting consumer payment behavior. In this post I will discuss how well credit scores will work for consumer lenders during and after the COVID-19 crisis and offer some recommendations for what lenders can be doing to measure and manage that model risk in a time like this. Perhaps no analytics innovation has created opportunity for more individuals than the credit score has. The first commercially available credit score was developed by MDS (now part of Experian) in 1987. Soon afterwards FICO® popularized the use of scores that evaluate the risk that a consumer would default on a loan. Prior to that, lending decisions were made by loan officers largely on the basis on their personal familiarity with credit applicants. Using data and analytics to assess risk not only created economic opportunity for millions of borrowers, but it also greatly improved the financial soundness of lending institutions worldwide. Predictive models such as credit scores have become the most critical tools for consumer lending businesses. They determine, among other things, who gets a loan and at what price and how an account such as a credit line is managed through its life cycle. Predictive models are in many cases critical for calculating loan and loss reserves, for stress testing, and for complying with accounting standards. Nearly all lenders rely on generic scores such as the FICO® score and VantageScore® credit score. Most larger companies also have a portfolio of custom scorecards that better predict particular aspects of payment behavior for the customers of interest. So how well are these scorecards likely to perform during and after the current pandemic? The models need to predict consumer credit risk even as: Nearly all consumers change their behaviors in response to the health crisis, Millions of people—in America and internationally—find their income suddenly reduced, and Consumers receive large numbers of accommodations from creditors, who have in turn temporarily changed some of their credit reporting practices in response to guidelines in the federal CARES Act. In an earlier post, I pointed out that there is good reason to believe that credit scores will tend to continue to rank order consumers from most likely to least likely to repay their debts even as we move from the longest economic expansion in history to a period of unforeseen and unexpected challenges. But the interpretation of the score (for example, the log odds or the bad rate) may need to be adjusted. Furthermore, that assumes that the model was working well on a lender’s population before this crisis started. If it has been a long time since a scorecard was validated, that assumption needs to be questioned. Because experts are considering several different scenarios regarding both the immediate and long-term economic impacts of COVID-19, it’s important to have a plan for ongoing monitoring as long as necessary. Some lenders have strong Model Risk Management (MRM) teams complying with requirements from the Federal Reserve, Federal Deposit Insurance Corporation (FDIC), the Office of the Comptroller of the Currency (OCC). Those resources are now stretched thin. Other institutions, with fewer resources for MRM, are now discovering gaps in their model inventories as they implement operational changes. In either case, now’s the time to reassess how well scorecards are working. Good model validation practices are especially critical now if lenders are to continue to make the sound data-driven decisions that promote fairness for consumers and financial soundness for the institution. If you’re a credit risk manager responsible for the generic or custom models driving your lending, servicing, or capital allocation policies, there are several things you can do--starting now--to be sure that your organization can continue to make fair and sound lending decisions throughout this volatile period: Assess your model inventory. Do you have good documentation showing when each of the models in your organization was built? When was it last validated? Assign a level of criticality to each model in use. Starting with your most critical models, perform a baseline validation to determine how the model was performing prior to the global health crisis. It may be prudent to conduct not only your routine validation (verifying that the model was continuing to perform at the beginning of the period) but also a baseline validation with a shortened performance window (such as 6-12 months). That baseline validation will be useful if the downturn becomes a protracted one—in which case your scorecard models should be validated more frequently than usual. A shorter outcome window will allow a timelier assessment of the relationship between the score and the bad rate—which will help you update your lending and servicing policies to prevent losses. Determine if any of your scorecards had deteriorated even before the global pandemic. Consider recalibrating or rebuilding those scorecards. (Use metrics such as the Population Stability Index, the K-S statistic and the Gini Coefficient to help with that decision.) Many lenders chose not to prioritize rebuilding their behavioral scorecards for account management or collections during the longest period of economic growth in memory. Those models may soon be among the most critical models in your organization as you work to maintain the trust of your accountholders while also maintaining your institution’s financial soundness. Once the CARES accommodation period has expired, it will be important to revalidate your models more frequently than in the past—for as long as it takes until consumer behavior normalizes and the economy finds its footing. When you find it appropriate to rebuild a scorecard model, consider whether now is the time to implement ethical and explainable AI. Some of our clients are finding that Machine Learned models are more predictive than traditional scorecards. Early Experian research using data from the last recession indicates this will continue to be true for the foreseeable future. Furthermore, Experian has invested in Research and Development to help these clients deliver FCRA-compliant Adverse Action reasons to their consumers and to make the models explainable and transparent for model risk governance and compliance purposes. The sudden economic volatility that has resulted from this global health crisis has been a shock to all organizations. It is important for lenders to take the pulse of their predictive models now and throughout the downturn. They are especially critical tools for making sound data-driven business decisions until the economy is less volatile. Experian is committed to helping your organization during times of uncertainty. For more resources, visit our Look Ahead 2020 Hub. Learn more